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How To Read This Book 



When Irving Sato and David Jackson first as>ked nne to prepare this 
Guidebook, I was really excited because it provided me with an opportunity 
to write about nny combined interests in gifted education and program evaluation 
But the initial excitement quickly gave way to a mild case of apoplexy as I 
began to gather together the resources that I would iieed for carrying out this 
assignment. In recent years, the literature on evaluation has multiplied a 
thousand-fold and the renewed interest in education for gifted and talented 
students has resulted in many new additions to the already voluminous 
literature in this area. I didn't want the Guidebook to be a replication or 
regurgitation of the many fine books and articles that have been written on 
prograng^evaluation; and yet there were certain basic ideas and concepts 
from the literature that seemed necessary for a full understanding of the 
evaluation game. The problem was further complicated by the diverse 
backgrounds and different types of people who we anticipated would be 
using the book. 

After rejecting the idea of running away to Tahiti, I decided to structure 
the Guidebook in a way that would serve several audiences, and at the same 
time, not require any individual to labor through the entire book. An attempt 
has been made to achieve this noble goal by dividing the content into chapters 
that answer particular questions for particular audiences. Thus, a person who 
"has had a great deal of experience in evaluation work, but limited experience 
in programs for the gifted, can skip certain-sections because they contain basic 
information with which he or she no doubt is familiar. This same person, 
however, may need to learn about some of the problems that are unique to 
evaluating programs for the gifted, and thus certain sections of the Guidebook 
will be relevant to his or her needs. 

Before discussing the questions that each chapter attempts to answer, I 
would like to say a few words about (1 ) the'five intended audiences of this 
book, and (2) a reference system that has been used throughout the book. 
The first audience might be best described as the would-be field evaluator. 
Although I envision this person as having had some background in measurement 
and evaluation, he or she is not typically a person who has done advanced 
work in this field and generally would not consider himself or herself to be 
an evaluation specialist. If a professional evaluator is not available, I consider 
the would-be evaluator to be a person within a school system or agency who 
will end up with the responsibility of designing and carrying out an evaluation 
study. Although it is hoped that the Guidebook \n\\\ provide this person with 
enough information for carrying out most of the job, it would be naive to 
assume that all of the necessary information and resources he or she will 
need are contained herein. It is mainly for this person that I have developed 
the Basic Reference System which is described below. The Guidebook cannot 
possibly make the would-be evaluator a specialist in psychometrics, statistics, 
or research design. These are highly technical areas to which professionals 
have devoted entire careers. But, because we need expertise from these areas 
to carry out quality evaluations, I have tried to suggest the specific points at 
which the would-be evaluator should seek the assistance of specialists. 
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The second audience for whom the Guidebook has been designed is 
referred to in the following chapters as program people or program personnel 
By this I mean persons within the school system or agency who are responsible 
for selecting an evaluator, making the evaluator aware of the program under 
consideration, developing an evaluation contract or agreement, and monitoring 
the work of the evaluator. The Guidebook will be most useful to this person 
or group because it attempts to provide a basic understanding of the problems 
of evaluating programs for the gifted, the options and alternatives that might 
be explored in developing an evaluation study, and the types of information 
they can expect to receive from an evaluation. For program people, communication 
is an essential responsibility — they must know enough about evaluation to 
negotiate meaningfully with a professional evaluator and to make certain that 
the evaluator serves their needs and answers their questions. 

The third group of persons who may have an interest in the Guidebook 
consists of state department personnel and policy makers who are responsible 
for developing guidelines for the evaluation of programs for the gifted and 
talented. Unless persons who write guidelines are aware of the unique nature 
of programs for the gifted as they relate to designing and carrying out evaluation 
studies, evaluations may end up placing unreasonable and sometimes even 
restrictive demands on programs. I am familiar with cases where evaluations 
turned out to be nothing more than a mechanistic compliance with poorly 
conceived guidelines, and in these cases the evaluations served limited purposes 
in program improvement. 

The fourth group of persons may, in certain cases, fulfill the same role 
as the group mentioned above. I n this case, I have in mind state department 
consultants in the area of education for the gifted who are not ordinarily 
experts in the field of evaluation, but who frequently must provide the 
technical assistance necessary so that people at the local level can evaluate 
their own programs or negotiate with professional evaluators. Hopefully, the 
Guidebook will provide these persons with the know-how to render technical 
assistance and to monitor the quality of evaluation studies carried out within 
their respective states. 

Finally, and at the risk of being somewhat presumptuous, I believe that 
the Guidebook will be of some-value to the professional evaluator who has 
not had a great deal of experience in working with programs for the gifted 
and talented. To the professional evaluator, some of the sections in Jhe 
Guidebook wilTbe very elementary, but other sections hopefully will make 
him or her aware of the unique problems with which we are faced in working 
with gifted and talented youngsters. 

In the past decade, more has been written on evaluation and accountability 
than perhaps any other single topic in education. One of the problems that 
I faced in preparing this Guidebook was how to provide the reader with a list 
of necessary references and background material. It is always easy for an 
author to throw in tons of references on particular topics, but this approach 
is not very economical in terms of the reader's time. I have tried to solve this 
problem by carefully screening the literature and selecting a relatively small 
number of basic references that strike right at the heart of the issues that will 
be discussed. The basic references, placed in boxes at points where I believe 
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they are highly relevant, are usually the most authoritative sources of information 
about particular topics or the most understandable treatment of highly technical 
information. Most of the basic references are relatively short and to the point. 
In cases where entire books have been recommended, they are usually source- 
books or reference books that are not intended to be read from cover to cover. 
In addition to the basic reference system, four topical bibliographies have 
been included that contain selected references in the following areas: 

A. Evaluation: Issues, Models, and Methods 

B. Instrument Source Books (Annotated) 

C. Instrument Construction 

D. Measurement, Statistics, and Research Design 

The first chapter attempts to answer questions that will be raised about 
why we should conduct an evaluation and what are some of the special 
considerations that must be taken into account when we are evaluating 
programs for the gifted and talented. This chapter can be used as a rationale 
for planning evaluation studies and as a guide for the evaluator who has had 
limited experience in gifted education. 

The second chapter contains basic information about evaluation models 
and concepts. Persons who are beginners in the area of evaluation will-find 
this chapter helpful in learning the jargon of the trade and the language through 
which evaluators communicate with one another. The models described in 
Chapter Two represent the best known systems developed to date and the 
only systems developed specifically for programs for the gifted. Persons with 
extensive experience in evaluation will probably want to skip this chapter 
because it is essentially a summary of very well-known models and concepts. 

The third chapter describes a particular system of evaluation that 
represents a synthesis of my experience in working with programs for the 
gifted and talented. Although it is not quite a cookbook, it does lay out very 
specific steps that can be followed in planning and carrying out an evaluation. 
A major part of this chapter deals with the always troublesome task of selecting 
and/or constructing appropriate instruments. This chapter will, in all probability, 
be of most value to the would-be evaluator who must walk into work one 
morning and actually begin to conduct an ey.3l.y|tion study. It is intended to 
be as practical as one can get in a guidebook that might be used for a variety 
of programs. 

The final chapter deals with some of the business aspects of evaluation. 
It points out issues that should be considered in selecting an evaluator, developing 
evaluation proposals, and negotiating evaluation contracts. This chapter is 
intended to help program people strike a good deal with an evaluator; and at 
the same time, it is intended to help the evaluator arrive at agreements that 
may be necessary for him to carry out his responsibilities. 

A Note on Point-of-View 

A guidebook of this type will quite obviously reflect certain beliefs and 
points-of-view on the part of the writer. There are essentially three areas in 
which I have expressed opinions that may not be universally shared by other 
persons writing in the field of evaluation. These points-of-view are based on 
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my experience in working with progranns for the gifted and talented and the 
problenns that 1 have encountered as a front-line evaluator in this area. 

First, I ann not a strong advocate of the traditional pretest/post-test 
approach to evaluation so far as progranns for the gifted""are concerned, nor do 
I believe that the rigid behavioral-objectivies approach is especially appropriate 
for evaluating programs that focus on higher level objectives. In my opinion, 
these approaches have placed too much emphasis on evaluating students rather 
than the programs that should be serving students. Program evaluation is a 
broad concept that embraces many kinds of information. If^'e restrict our 
activities to test score analysis only, we may be guilty of judging the consumer 
rather than the service that has been provided to him or her. For reasons that 
will be pointed out in the chapters that follow, I firmly believe that people in 
gifted education must take the lead in breaking down the let's-give-a-lot-of tests 
mentality that has characterized (and plagued, in many cases) the field of 
evaluation for many years. The "accountability boys" are trying to force rigid 
models on programs that are straining to be free and open; and when this 
happens, the tail ends up wagging the dog. I believe that the role of evaluation 
is to serve programs rather than to force them, however subtly, into being 
convenient vehicles /for the evaluator's bag of tricks. 

My second bias is that regardless of who hires and pays the evaluator, he must 
have the best interests of the consumer (student) at heart-he must view himself 
as an evaluator for the people rather than for the institution that provides the 
services to people. While we cannot be politically naive in designing evaluation 
studies, we also should never forget that an honest evaluation is impossible unless 
the best interests of the consumer are first and foremost in our minds. 

Finally, I believe that evaluation is basically a simple process that need not be 
shrouded in complicated language, statistics, or the jargon of psychometrics. 
When we blow away the nostalgic dust that surrounds the evaluation mystique, 
we find an essentially logical process that.should not be over-elaborated upon, 
nor should it be an end in and of itself. People usually raise simple, straightforward * 
questions and the evaluation should attempt to provide simple, straightforward 
answers. 
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CHAPTER ONE: Evaluation! Who Needs It? 



I. The Need for Program Evaluation — A Matter of Support and Survival 

Did it do any good? 

Was it worth the time and effort? 

Was it worth the nnoney? 

Is it working as we expected? 

Is this approach better than sonne other nnethod? 

These are questions that are bejjig asked with increased frequency as 
schools pursue what sonnetinnes seenns to be the innpossible dreann of providing 
each student with the very best possible learning environnnent for his or her 
particular needs. Although special provisions for the gifted and talented are 
an essential part of any school progrann that truly respects the principle of 
individual differences, the connpetition for linnited resources annong all types 
of supplennentary progranns frequently causes the needs of the gifted to be 
relatively low on the list of educational priorities. When school budgets are 
"cut" it is not unusual for the gifted program to be one of the first items to 
be eliminated. And when boards of education, legislatures, and other sponsoring 
agencies review the many special programs for which they are called upon to 
support, the very survival of programs for the gifted may depend on having 
evaluation information readily available. All too often, evaluations have been 
launched as last-ditch efforts to save programs that are in danger of being 
eliminated or sharply reduced in the amount of support they receive from 
sponsoring agencies. Although a hastily conducted evaluation may be better 
than no evaluation at all, the best weapon in the battle for program support 
and survival is a carefully planned and comprehensive evaluation that will 
accurately document all aspects of the services being provided for gifted 
and talented youngsters. Evaluation should bean essential and ongoing part 
of total programming and each step of the planning and development phases 
of a program for the gifted should give careful attention to the ways in 
which evaluative information can be gathered, organized and presented to 
decision making individuals or groups. 

The need for program evaluation in gifted education has grown out 
of a general concern on the part of decision makers for greater accountability 
in all aspects of education. In the past, innovation in education and especially 
efforts to help youngsters with unusual needs such as the gifted and talented 
were looked upon with a strangely philanthropic attitude. We accepted the 
notion that innovative efforts equaled innovation itself — that sincere and 
honest attempts to improve the education of gifted students were de facto 
indicators of favorable results.. In other words, the.attitude of "trying equals 
success" often caused us to minimize the need for program evaluation and, 
indeed, this attitude sometimes served as a substitute for evaluation. The 
person who was bold enough to raise serious questions about the value or 
quality of a particular program was frequently looked upon as some sort 
of malcontent, especially if the program in question was cloaked in the mantle 
of innovation, launched with great fanfare, and happened to be the "brain child" 
of an influential group or well-known "expert" in the educational establishment. 
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Programs for the gifted have been especially vulnerable to substituting 
the "trying equals success" attitude for rigorous attempts to evaluate program 
effectiveness. Innovative approaches to learning are frequently used with gifted 
youngsters and many programs for highly able students are characterized by 
"flashy" activities, materials, and student products. If you want to see "great 
things" happening in a school, visit the classroom or center for the gifted. 
If you want to see outstanding short stories, or poetry, or science projects 
look at those that have been produced by gifted and talented youngsters. 
Gifted students, because they are gifted, have the capacity for high level 
performance and their products are often of superior quality. But the nagging 
questions the evaluator must always examine are: "What type, of, programmatic 
learning experiences fostered this outstanding performance? Was the performance 
really attributable to the program or did it emerge simply because the child 
was gifted and we had the good sense to stay out of his way?" 

In recent years the unprecedented expansion of innovative approaches 
to learning and the rapid development of educational technology have provided 
educators with a wide variety of alternatives that can be used in programming 
for students with special needs. But the availability of these alternatives has 
also given rise to many questions that can only be answered through systematic 
program evaluation. These questions generally deal with the appropriateness 
and effectiveness of various learning materials, teaching strategies, and approaches 
to program organization and management. Fortunately, the technology in 
educational measurement and evaluation has also undergone a period of increased 
growth and development, the greater sophistication and availability of 
evaluation methods has provided us with some of the tools necessary for 
documenting the value and effectiveness of programs for the gifted. The purpose 
of this book is to summarize some of the models, materials, and techniques 
that can be used by persons who are charged with the responsibility of 
evaluating programs that serve gifted and talented youngsters. 

II. The Purpose cf Program Evaluation 

A. The Importance of Decision Making 

The general purpose of evaluation is to gather, analyze, and disseminate 
information that can be used to make decisions about educational programs. 
Evaluation should always be directed toward action that hopefully will 
result in the improvement of services to students through the continuation, 
modification, or elimination of conditions which effect learning. 

It should be emphasized that the conditions which effect learning are 
not necessarily restricted to the instructional process. For example, if 
admission to a special program ^or artistically talented youngsters is largely 
dependent on IQ test scores, the evaluator may want to investigate the 
policies underlying the identification system and the effectiveness of the 
screening and selection procedures. 

Economy and efficiency can be improved in an evaluation design if 
we begin by raising three interrelated questions: 

1 . Who are the decision makers at various levels of possible action? 

2. Over what actions do decision makers have control? 

3. What information is necessary for making decisions? 




Perhaps the best vvay to illustrate the interrelatedness and the importance 
of these questions is by developing a hypothetical example. Let us suppose 
that we are evaluating an anthropology program for gifted students in a 
junior high school. The program involves several trips to various city dumps' 
where students attempt to study differences between communities by 
analyzing the types of objects people discard. Supplementary expenditures 
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are necessary for transportation, insurance, reference books, and a consultant 
in anthropology. Who are the decision makers and over what actions do they 
have control? The board of education must approve the supplementary funds 
and therefore continuation of the program (action) rests with them. Let us 
speculate that one question with which the board is concerned is whether 
or not parents are satisfied with the program. One segment of the information 
necessary for decision making thus becomes parental attitudes. This information 
might be gathered through the use of a questionnaire and/or interviews with 
a random sample of parents. 

Another decision maker in this situation might be the school principal. 
It may be up to him or her to decide when the trips take place and whether 
or not any problems are resulting because students must be excused from 
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some of their regular classes. The principal may require information about 
student performance and conscientiousness in completing regular classroom 
work that they may have missed as a result of participating in the special 
program. This information will help determine if the program should take 
place during school hours or at some other time. It may also reveal some 
unexpected findings such as hostility toward the program on the part of 
certain teachers, in this case the evaluator might want to recommend that 
greater efforts be made to familiarize the general staff with the nature of 
the program, why it is being provided for gifted students, and how the 
program fits into the general philosophy and objectives of the total 
school program. 

Finally, the teacher and consultant in anthropology are decision makers 
in this situation because they have control over actions which relate directly 
to possible modifications in the instructional process. The/ may require 
information about the students' knowledge in anthropology and their ability 
to analyze data in social science. The teacher may also want to obtain 
information about student satisfaction with the program and his or her own 
success in working with gifted youngsters. Thus, tests, student questionnaires, 
and rating scales will be needed to help the teacher and consultant make 
decisions about how they might like to modify the instructional process. 

The interrelationship between evaluation, decision makers, and alternative 
types of programmatic action are depicted in Figure 1 . Although this figure 
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points out the influence of .evaluation information on decision makers, it 
would be naive to assume that subjective values and attitudes do not also play 
an influential role in this process. When planning a program evaluation, efforts 
should be made to gain an awareness of the personal and organizational values 
that may influence important decisions. Such an awareness will enable the 
evaluator to gather and perhaps emphasize types of information that might 
be influential in combating a negative attitude toward a program or a particular 
aspect of a program. For example, if a school board member(s) feels that 
programs for the gifted encourage "elitism" on the part of participating students, 
the evaluator may want to place added emphasis on this component of his 
investigation. Although values and attitudes are difficult to change, an 
awareness of their existence will at least give the evaluator an opportunity 
to seek out information that might be influential in modifying attitudes. 

By way of summary, decision making is a fundamental goal of evaluation 
and therefore it is important to identify decision makers and the actions 
over which they have control at the beginning of any evaluation endeavor. 
Since the structure and focus of an evaluation should be guided by the 
decision-making process, it is recommended that this process be analyzed 
during the planning phase of an evaluation. The evaluator and decision 
makers should work closely together in an attempt to identify the parameters 
of change over which each decision making individual or group has control. 
A discussion dealing with analyzing the decision-making process is presented 
in Chapter III. It should be pointed out that some changes may be under the 
control of more than one decision-making individual or group. However, an 
analysis of the areas of responsibility will help to clarify contingent relationships 
among decision makers. 
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A thorough treatment of the decision-making process and decision- 
making models is presented in: 

Stufflebeam, Daniel L.,efa/. Chapter 3: "Educational Decision 
Making," in Educational Evaluation and Decision Mal<ing, 
Itasca, Illinois: F. E. Peacock Publishers, Inc., 1971, pp. 
49-105. 



B. The Objectives of Program Evaluation 

Within the general decision-making purpose of program evaluation discussed 
above, there are a number of more specific objectives which help to give direction 
to the actual design of an evaluation. At this point, however., it is important to 
emphasize that the purpose of an evaluation is NOT to come up with a simple 
score or rating that attempts to express the success or failure of a given program. An 
evaluation is scarcely worth the paper it is written on if it does not provide relatively 
specific information that supports the maintenance, modification, or termination 
particular program components. Thus,.an evaluation should be "diagnostic" in 
the sense that it pinpoints by careful examination the circumstances and conditions 
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that result in identifiable changes in performance, attitude, or other indicators of 
progrann effectiveness. In order for an evaluation to play a constructive and 
posftive role in the overall process of '^(ii'i^'cation, it should attennpt to fulfill as 
many of the following objectives as possible: 

1 . To discover whether and how effectively the objectives of a program 
are being fulfilled. 

2. To discover unplanned and unexpected consequences that are 
resulting from particular program practices, 

3. To determine the underlying policies and related activities that 
contribute to success or failure in particular areas. 

4. To provide continous in-process feedback at intermediate stages 
throughout the course of a program. 

5. To suggest realistic, as well as ideal, alternative courses of action 
for program modification. 

The first of these objectives is considered by many persons to be the major 
responsibility of the evaluator. The stated objectives of a program generally 
reflect the educational values and attitudes underlying a program. Although it is 
important to determine the degree to which each objective is being achieved, it is 
equally important for the evaluator to discover both positive and negative outcomes 
that are essentially unplanned and, therefore, not stated in the objectives. For 
example, in an evaluation of a special program for gifted high school students it was 
found that the program was influential in helping a large number of students to 
clarify their career choices. A small number of students also reported that their 
involvement in the program had caused them to give up the use of marijuana! While 
neither of these outcomes were included in the stated objectives of the program, 
they nevertheless provided powerful support for the continuation of the program. 

The distinction between stated objectives and anticipated outcomes on one 
hand, and unexpected and unplanned consequences on the other, raises a serious 
theoretical issue in program evaluation. Briefly and simply stated, this issue 
revolves around whether the evaluator should have the right and obligation to 
investigate all aspects of a program or whether his work should be restricted to 
an assessment of stated objectives. Although most contemporary evaluation 
theorists would agree that measuring the attainment of objectives is the most 
important goal of an evaluation, they would also support the position that the 
evaluator must be free to investigate any and all conditions that may influence 
the effective operation of a program. If the evaluator is told "where to look'' and 
"what to look at" he may very well overlook important factors contributing to 
the success or failure of particular aspects of a program. Jherefore, the position 
taken here is that any activity or condition which may have a direct or indirect 
impact on a program is "fair game" for the evaluator. 

The third objective of program evaluation listed above— to determine underlying 
policies and related activities that affect a program— calls attention to the fact 
that a successful program is frequently the result of policy decisions and actions 
that may influence instruction, but are not a direct result of instruction itself. 
For example, the procedures by which teachers are selected for a program may be 
based on a policy underlying teacher selection. This policy may be formal and 
written or it may simply exist in the minds of persons who are responsible for 
selecting teachers. But nevertheless, the policy (or lack of a policy) could have a 
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serious impact on the program. Take, for example, a program in which teachers 
are selected on the basis of longevity or seniority rather than superior teaching 
ability. This policy may result in the haphazard selection of teachers, and if 
ineffective teachers are chosen the program may be doomed to fo'lure long before 
students enter the classroom. 

Other non-instructional factors, such as planning and program management 
should also be evaluated. The following example taken from an actual evaluation 
illustrates how non-instructional factors may influence a program. In the • 
evaluation of a six-week summer program was found that the supplies and 
materials did not arrive until the beginning of the fourth week of the program. 
Upon further investigation it was determined that this condition was due to 
poor planning and had, in fact, happened two years in a row. Although the 
tardiness of supplies and materials certainly influenced instruction, the major 
recommendation growing out of this finding clearly related to planning and 
management on the part of the project diector. (This finding, coupled with other 
consequences of poor management, ultimately led to the dismissal of the 
project director.) 

The fourth objective of program evaluation— to provide continuous 
feedback throughout the course of a program— calls attention to one of 
the basic distinctions between evaluation and research. This distinction 
is concerned with the responsibilitiy for suggesting changes in program 
activities while the program is in progress. Generally, the researcher is 
interested in judging the effectiveness of a predetermined, carefully 
controlled, and relatively specific "treatment." He is concerned with the 
-generalization and replication of a prescribed activity, and therefore he 
is obligated not to interfere or make suggestions while the prescribed 
activity is taking place. The evaluator, on the other hand, is concerned 
with program improvement and he is obligated to provide continuous 
feedback so changes and modifications can be made as a program progresses. 
Just as a ship's captain can miss his final destination by a wide margin if 
he fails to make small navigational corrections when his boat is off course 
by a few degrees, so also can a program "miss" its goal if the director 
does not recognize problems when they occur and take whatever action 
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may be necessary to correct them. Continuous monitoring will help to nip many 
problems in the bud and avoid the kind of dissatisfaction that builds up when 
even the smallest problem is left unattended. Evaluators who employ the 
researcher's traditional pre-post measurement design without providing in-process 
feedback may be guilty of letting students "suffer through" experiences that 
ultimately will reflect negatively on the program in general. 

The final objective listed above— tcf suggest realistic as well as alternative 
courses of action— is concerned with the usefulness of evaluative findings. 
Realistic suggestions are those that take into consideration such predetermining 
conditions as the availability of funds and human resources, the prevailing 
political climate, attitudes toward certain types of programming on the part of 
key decision makers, and how a program for the gifted can work harmoniously 
within the overall framework of a particular school or school system. There 
would be little value in making a recommendation that requires a $100,000 
outlay if the evaluator knows full-well that this money is not available. Similarly, 
an evaluator is being unrealistic if he recommends that an entire school system 
change its scheduling procedures because it is not meeting the needs of the 
gifted. Such recommendations are evaluation "cop-outs" unless the evaluator 
also suggests some alternatives that are within existing resources or within the 
realm of possible action on the part of decision makers. 

To summarize, there are a number of objectives that can serve as guides to 
a comprehensive and systematic evaluation. An evaluative study that achieves 
all of the objectives discussed above will undoubtedly provide the kind of 
information necessary for decision making. 

III. Special Problems in Evaluating Programs for the Gifted and Talented 

A. The Problem of "Higher Level" Objectives 

Programs for the gifted are often characterized by a commitment to the 
development of higher powers of mind and advanced levels of awareness, interest, 
and other affective behaviors. This presents a somewhat unique problem for the 
evaluator because these objectives cannot be measured as easily and precisely 
as those objectives which deal mainly with the acquisition of basic skills. As we 
move up the scale of learning behaviors, from the simple acquisition of knowledge 
to the development of higher mental processes, it becomes increasingly difficult 
to find measuring instruments that meet the scientific and practical requirements 
necessary for good evaluation studies. While virtually dozens of relatively valid 
and reliable tests are available to measure skills in the traditional areas of school 
avhievement, instruments for evaluating higher level processes of objectives are 
not so readiiy available. I n areas where these instruments have been developed 
(e.g., the Torrance Tests of Creative Thinking) they are often expensive to 
administer and/or score, and therefore their use in an evaluative study may be 
economically unfeasible. 

A second dimension of this problem is that gifted programs are frequently 
characterized by highly individualized objectives for each student. Whereas a 
reading skills program for average or slow learners may have enough uniformity 
in its objectivies to warrant large scale standardized testing, a program for gifted 
students may have many different objectives for each student. The reliability of 
most standardized tests is a function of group size and it is extremely difficult to 
show statistically significant pretest to post-test gains when only a few students 
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are being evaluated with a given instrument. Standardized tests can, of course, be 
effectively used in evaluating progranns for the gifted if they (a) are valid 
(appropriate) measures of particular objectives, and (b) if they are used in 
situations where reasonable levels of reliability can be obtained. But when a 
teacher devises individualized objectives for each child, as is often the^ease iTT 
programs for the gifted, then we must seriously question the appropriateness of 
tests based on systemwide or nationwide objectives. 

In recent years there has been a great deal of concern in education about 
the specification of objectives in terms of observable and measurable student 
behaviors. Many evaluators have looked upon the "behavioral objectives model" 
as a panacea for conducting evaluation studies. However, the nature of gifted 
programs and their concern for developing higher thought processes may make 
this model too cumbersome to be practically applied to programs for the gifted 
and talented. In fact, when the behavioral objectives approach is used in its 
most rigid form, it may even force program developers to focus their attention 
on the trivial rather than important behaviors of superior learners. 

The rigid behavioral objectives model is inappropriate for programs for the- 
gifted because it forces us to be primarily concerned with those behaviors that 
are easily measured. Such a situation may result in the tail wagging the dog. 
Michael Scriven, the single-most influential person writing on educational 
evaluation today, has pointed out that "putting pressure on (a person) to formulate 
his goals, to keep to them, and to express them in testable terms may enormously 
alter his product in ways that are certainly not always desirable" (Scriven, 1967, 
p. 55). Here is an example of a rigid objective taken from a book written by a well- 
known object ivist: 

At the conclusion of three hours of instruction at least 75 per- 
cent of the students will be able to deliver a 15 minute 
extemporaneous speech violating no more than two of the 
twelve "rules for oral presentation" supplied in class as judged 
by a panel of three randomly selected classmates using the 
standard rating form (Popham, 1972, p. 16). 




ANP WE NEVE^ mmj HE'P 

MAJCE IT H\meH SPBEC** OA^. 
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Let us raise a few questions about this objective: 

Suppose some students need more or less than three hours 
of instruction? 

Suppose some students would rather formulate their own 
rules for oral presentation for a variety of speaking situations? 

Suppose some students don't like to give extemporaneous 
speeches? 

And suppose, heaven forbid, some students don't give a 
darn about the rules for oral presentation? (I wonder 
what Eric Hoffer, or Will Rogers, Bob Dylan or Stokely 
Carmichael would think about the "rules for oral presentation.") 

And finally, are we making the best use of all children's 
time when we force them to listen to and fill out rating forms 
on extemporaneous speeches that other children have been 
forced to make — extemporaneously, of course? 

Perhaps this model is appropriate for certain kinds of learning in the basic 
skill areas, but as far as gifted students are concerned, it may be putting a 
strait-jacket on the types of learning experiences which they will be allowed 
to pursue. Thus, it is the very nature of the objectives of gifted programs that 
make the rigid behavioral objectives model inappropriate. In gifted education 
we talk about some pretty "high-falutin" objectives; objectives such as creativity 
and critical thinking and problem solving. We talk about types of learning that 
may lead to analyzing a moral principle, or synthesizing a political argument, 
or evaluating a philosophic point of view. We talk about producing unique 
plans, communication forms, and ideas. And we also talk about some relatively 
sophisticated non-cognitive objectives-such as developing favorable attitudes 
toward learning, or developing acceptance of and even appreciation for opposing 
points of view, or showing a commitment to a cause by taking affirmative 
actions. Although many experts in the testing business believe that these complex 
objectives can be evaluated, Robert Stake, one of America's foremost authorities 
on evaluation (Stake, 1973, p. 199), has suggested that the total cost of 
measuring such objectives may be one hundred times that of administering 
a forty-five minute standardized paper-and-pencil test; and the amount of time, 
personnel, and facilities necessary for such evaluation may be astronomical. 
Stake also points out (Stake, 1973, pp. 196-199) that the errors of testing- 
increase markedly when we move from highly specific areas of performance 
to items which attempt to measure higher mental processes and unreached 
human potential. According to Stake, the only reason we have tolerated the 
test error in standardized instruments is because very few important educational 
decisions are ever based on test scores alone. 

Stake goes on to point out that the behavioral objectives model of program 
evaluation become very popular as part-and-parcel of the performance contracting 
movement — an ill-fated attempt "to substitute technical procedures for . 
personal attention" (Stake, 1973, p. 194) in teaching, especially teaching 
directed toward remediation among so called disadvantaged students. The 
many problems which have beset the performance contracting movement are 
well documented in the literature on measurement and evaluation, and for these 
reasons we should be very cautious in considering the behavioral objectives 
evaluative model for gifted education. 



B. Measurement and Statistical Problams 

Measurement and formal testing often play a major role in evaluation 
studies, but certain cautions are necessary when we consider the use of 
standardized tests in evaluating programs for the gifted. In addition to the 
measurement problems implicit in the above discussion on "Higher Level 
Objectives," problems often arise when we attempt to use norm referenced 
tests developed for general populations. Conventional standardized tests are 
based on the normal distribution curve and for this reason the equality of 
units of measurement is open to serious question. The main issue in using 
age, grade, or percentile norms is that we cannot assume that a year's growth 
or growth in a given number of percentile points is a uniform unit. Thus, 
for example, if the performance of an average student increases from the 
40th to the 50th percentile over the course of a school year, we cannot 
assume that this is a greater gain than that made by a gifted student whose 
score increased frorn the 90th to the 95th percentile. The gifted student 
initially scored at the upper end of the normal curve where it is much more 
difficult to show an increase in percentile score points. The same is true for 
age and grade scores. Generally, there is a slowing down of gains at the upper 
levels of most performance tests that were normed on the general population. 
For this reason, when the evaluator uses standardized tests, he should avoid 
making comparisons between gifted students and other populations. This 
can be done by developing separate sets of norms for each distinct population 
whose growth is being evaluated, provided of course, that the test has a broad 
enough range to allow students to show maximum growth. If a test does not 
have enough "top" in it, highly able students may score at the upper limits, 
but we will be unable to determine their true growth because of the low ceiling 
of the test. Since many standardized tests are designed to provide achievement 
information for the vast middle ranges of ability, their content and interpretive 
data may not be valid for children who deviate markedly upward from the mean. 



BASIC REFERENCE on Inequality of Norms 



A thorough and yet highly readable discussion of cautions 
in the use of norms can be found in: 

Thorndike, Robert L. and Elizabeth- Hagen. 

Chapter?: "Norms and Units for Measurements," in 
Measurement and Evaluation in Education and Psychology^ 

New York: John Wiley and Sons, Inc. 
1969, pp. 210-243. 



The use of conventional tests with gifted and talented students also 
presents some problems in the statistical treatment of evaluative data. As 
was pointed out earlier, test reliability is a function of group diversity— 
the more heterogeneous the group the higher the reliability. Since gifted 
groups frequently are, by definition, relatively homogeneous groups, and 
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therefore frequently show a narrower range of test scores than the population 
in general, we should be extremely cautious when viewing the reported 
reliabilities of standarized tests. Unfortunately, most test publishers do not 
report reliabilities for subpopulations within their standardization sample 
and therefore it may be necessary to conduct a "local" reliability study 
whenever conventional tests are used with special populations. 

One of the major statistical problems encountered when working with 
the test scores of superior students is the well-known "regression toward the 
mean" effect. Although this is a complicated statistical phenomenon, simply 
stated it means that predicted scores tend to "move in" toward the mean of 
the distribution. Thus, if we are using a pretest and post-test design to evaluate 
the effects of a program for the gifted, and it the students' scores on the 
pretest are initially high, it is quite likely their post-test scores will actually 
decrease due to the regression effect. It is for this reason the evaluator must 
exercise a great deal of caution when considering the pre/post design and 
other statistical designs that do not take into account the lack of normality 
in the distribution of gifted students' test scores. When pretest and post-test 
scores are used, it may be necessary to explore the use of nonparametric 
statistics or multivariate methods of analysis. 



BASIC REFERENCE on Problems in Evaluating Special Populations 



This article was written mainly for persons who are concerned with 
evaluating programs for the physically and mentally handicapped; 
but because it deals with populations that frequently do not conform 
to the normal distribution curve it is applicable in many respects 
to our concerns with gifted students: 

Jones, Reginald L, "Accountability in Special Education: 
Some Problems," Exceptional Children, 1973, 39, 631-642. 



C. Practical Problems 

The evaluation of programs for the gifted, like evaluation in all other 
areas, requires time, money, and trained personnel. When evaluation js 
"tacked on" to a program as an afterthought, and when the human and 
financial resources necessary for carrying out a comprehensive evaluation 
are not available, the evaluator may very well end up being asked to do. 
the impossible. Even when time and resources are available, the evaluator 
is frequently called upon to develop a plan of evaluation for programs with 
poorly defined objectives and a very limited conception about what will 
constitute a successful program. Although these are practical, problems 
they can, nevertheless, have as much influence on the value and quality 
of an evaluative study as the measurement and statistical problems discussed 
above. Indeed, practical problems more often than not underlie or give rise 
to more complicated problems in measurement and design. 



23 



What can be done about practical problems in evaluation such as time, 
money, and personnel? The answer to this question often rests with persons 
who are responsible for drafting the guidelines for special programs and/or 
those persons who actually develop the programs. While it seems almost 
trite to say that provisions and resources for evaluation should be included 
or "written into" guidelines and proposals, the fact is that without such 
provisions evaluation becomes a game (sometimes even a farce) that can 
serve very limited purposes. Many evaluation specialists have now reached 
the point where they are simply saying that a program may be "unevaluatable" 
because program planners have not given serious attention and resources to 
the evaluation component. In Chapter Three of this guidebook, attention 
will be given to developing guidelines for evaluation that can be used in 
preparing state plans for programs for the gifted. 

Almost all writers in the field of evaluation have stressed the importance 
of involving the evaluator from the very beginning of any educational endeavor. 
Through such involvement, the evaluator can continually bring to the attention 
of program developers the steps that must be tai<en and the resources that 
must be allocated if evaluatiori is to serve useful purposes. Early and continuous 
involvement on the part of the evaluator will help to overcome many of the 
difficulties that arise when evaluation is tacked-on as an afterthought, but we 
would be naive if we did not recognize a practical problem that may arise as 
a result of the evaluator's influence. This might be described as "the-tail-wagging- 
the-dog" problem. In his efforts to develop a "respectable" evaluation design, 
the evaluator may "steer" a program in very subtle ways. For example, if the 
evaluator insists upon using certain standardized tests to assess the effectiveness 
of a particular program or practice, the teachers may very well epd up "teaching 
the tests,"or at least modifying their intended teaching activities because they 
feel the threat of evaluation hanging over their heads. There is no easy way 
to resolve the dilemma that often exists between the evaluator's need to be 
rigorous and scientific and the program developer's desire to'be innovative 
and flexible in programming for gifted and talented students. A basic guide, 
however, should be that the program determines the type of evaluation design 
and the instruments to be used, and that it is the evaluator's responsibility 
to respect the integrity of the program when he is planning his evaluation 
activities. This does not mean that the evaluator should not make suggested 
changes while the program is in process (See section on Formative Evaluation), 
but these suggestions should grow out of findings about the program as it 
exists in its "natural" form rather than as a result of the evaluator's influence 
on the program. 



BASIC REFERENCE on Problems and Issues in Evaluation 



A few years ago, I became particularly frustrated about the many 
problems associated with program evaluation. In sheer exasperation, 
I picked up my pen, dipped it in blood, and wrote the following 
paper. It points out some of the basic problems about which the 
evaluator should be aware. 

Renzulli, Joseph S. "The Confessions of a Frustrated Evaluator," 
Measurement and Evaluation in Guidance. 
1972,5,298-305. 
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Another practical problem relates to the attitude that many educators 
hold toward evaluation. Teachers and other professional personnel often 
view evaluation as a means of controlling or checking up on a program and 
the persons responsible for operating a program. In short, evaluation can be 
a very threatening affair that might result in some rather harsh actions, 
especially if the evaluation is mandated by a decision-making body or outside 
funding agency. 

In the early days of the evaluation business, we focused almost totally 
on. the measurement of growth in students, usually through the use of standardized 
achievement tests. To evaluate students was fine, but to evaluate ourselves was 
nothing less than educational heresy. "I .certainly wouldn't want to be held 
accountable," says the perceptive teacher, "if my class doesn't make sufficient 
gains on a standardized test. If their last year's teacher has prepared them 
poorly in areas that are measured by the tests, why should / be held accountable. 
After all, I'm only one pair of hands on the assembly line and if the end product 
isn't too good, you can't place all the blame on me." This is a favorite argument 
when people talk about teacher evaluation through student performance. And 
yet, we need the cooperation and assistance of teachers and other educational 
personnel if we are to obtain accurate and undistorted evaluative information. 
This is, indeed, a sticky problem for the evaluator who may, for example, end 
up in the position of asking a program director for cooperation in gathering 
evaluative data that might lead to criticism of the director or perhaps even result 
in the director's dismissal! 

Although negative or at least cautious attitudes toward evaluation are 
not easy to overcome, it is important for the evaluator to take steps toward 
dealing with this problem. The most obvious action that can be taken with 
regard to this problem is to create a positive atmosphere of helpfulness rather than 
destructiveness. The evaluator should point out that he wants to report the 
"good things" that are happening in a program just as much as those aspects 
of a program that are in need of change. He should also explain his role to 
teachers and administrators in terms of the ways in which he can help each 
particular group. He might, for example, say to teachers: "I can act as an 
omnibusman or intermediary between you and the decision makers by helping 
to make your needs and concerns known to the administration or the board 
of education." Unless persons being evaluated can see some value and benefit 
for themselves as a result of participating in an evaluative study, they are likely 
to approach the process halfheartedly; or even worse, they may actually try 
to distort evaluative information. 

The evaluator must walk a very thin line in the process of gaining the 
acceptance and cooperation of persons over whom he may eventually have to 
pass judgment. A friendly and cooperative atmosphere is very important; but 
at the same time, the evaluator should not forget that his major concern is 
how effectively a program is serving students. An evaluator who trades friendly 
relations with professionals for a primary concern for students may end up 
losing some of the "clout" that is necessary to bring about program improvement. 
In spite of all of the rhetoric about friendly and cooperative relations between 
evaluator and staff, the fact remains that the evaluator may sometimes have 
to recommend actions or changes unpleasant to the persons who sponsor and 
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POOR CHILD* 



College Professor- 
Such rawness m a pupil is a shame. 
Lack of Preparation in the High School is to blame, 



High School Teacher: 

Good Heavens, what crudity, the boy's a foot! 
The fault, of course, is in the Grammar School 

Grammar School Teacher: 

From such stupidity may I be spared. 
They send them up to me so unprepared. 

Primary Teacher: 

Kindergarten Blockhead! 
That preparation. Worse than none at all. 

Kindergarten Teacher: 

Such lack of training never did I see. 
What kind of woman must the Mother be! 

The Mother: 

Poor helpless child - He's not to blame. 
His father's folks are just the same. 



•Author Unknown 
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operate a program. This problem can be minimized by spelling out the 
responsibilities of the evaluator and the staff at the very beginning of an 
evaluation study. A section dealing with the development of an evaluation 
contract is included in Chapter Four. 



BASIC REFERENCE on Attitudes Toward Evaluation 



A very insightful and highly readable treatment of issues relating 
to attitudes toward evaluation has been presented in the following 
article: 

House, Ernest R. "The Conscience of Educational Evaluation/' 
Teachers College Record, 
1972,73,405-414. 



By way of summary, this section has dealt with some of the problems 
that might be encountered in evaluating programs for the gifted and talented 
Additional problems relating to particular approaches -.o e>^aluation will be 
discussed in the section of Chapter Two that deals with Evaluation Designs 
and Concepts. Although some of the problems are difficult to overcome, an 
awareness of their existence should help the evaluator to avoid falling into 
some of the traps that typically plague evaluation studies in this area of 
special education. 
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CHAPTER TWO: Evaluation Models and Concepts 

I. An Overview of Basic Evaluation Models 

This section contains a brief summary of five evaluation models. In most 
cases the models are extremely complex; and although the summaries are not 
intended to make you proficient in the use of any particular model, an awareness 
of the essential characteristics of each approach should enablo you to decide 
whether or not you would like to explore further one or more of the models. 
To assist you in further exploration, a Basic Reference has been provided at the 
end of each section. These references represent the primary source of information 
about each model. 

The first three models discussed below are general approaches to program 
evaluation that have been developed to assess various aspects of all types of 
educational programs. Although they were not developed specifically for 
evaluating programs for the gifted and talented, they can be used for this purpose 
because th v r organizational frameworks are broad enough to encompass a wide 
variety of educational activities. The last two models were developed specifically 
for programs for the gifted; but again, they are relatively broad in scope and 
therefore can be applied to many different types of special programs. Thus, the 
approaches described below should be viewed as multidimen.^ional systems within 
which the details of evaluation planning can be placed. 

A word of caution should, however, be offered before presenting the models. 
Many persons writing in the field of educational evaluation have recognized the 
models as mainly theoretical or heuristic (furthering the state of the art) contri- 
butions. As such, they provide us with several powerful ideas atoaf evaluation, 
and they offer useful suggestions for planning evaluative studies. But at the same 
time, the models may be inoperable as cookbook formulas for carrying out a 
total evaluation. A would-be evaluator could easily drown in a sea of complexity 
if he selected a single model and slavishly tried to stick to it. While each author 
has made a valuable contribution to the overall thinking about program evaluation, 
it is probably true that no single model will serve all of the evaluation needs of 
a given program. Because of differences in program structures, the availability of 
resources, and the general orientation of evaluators and decision-making bodies, 
it is recommended that the prospective evaluator review all of the models and 
then select the most useful concepts from each according to his particular, 
evaluation needs. 

A. Stake's "Countenance" Model 

Robert Stake characterizes the two basic acts of formal educational 
evaluation as description and judgment, both of which should be carried out 
to the fullest extent. There must be an effort to spell out antecedent conditions 
and classroom interactions, and to couple them with various program outcomes. 
Stake proposes that the evaluator consider three types of information, which he 
terms antecedent, transaction, and outcome data (See Figure 2.) Antecedent data 
are any data or conditions existing prior to the teaching and learning experience 
which may relate to the program outcomes. 
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The status of a student prior to his lesson, e.g., his aptitude, 
previous experience, interest, and willingness, is a complex 
antecedent. The programmed-instruction specialist calls some 
antecedents "entry behaviors." The state accrediting agency 
emphasizes the investment of community resources. All of 
these are examples of the antecedents which an evaluator will 
describe. (Stake, 1967) 



INTENTS OBSERVATIONS STANDARDS JUDGMENTS 



RATIONALE 



ANTECEDENTS 



TRANSACTIONS 



OUTCOMES 



DESCRIPTION MATRIX JUDGMENT MATRIX 



Figure 2 

A Layout of Statements and Data to be Collected by the Evaluator of an 
Educational Program. 



In addition to describing fully all the surrounding conditions and influences 
which may affect a program, the evaluator must also be a keen observer of trans- 
actions. Transactions are the countless encounters which comprise the process of 
education, the dynamic interactions between teacher and student, author and 
reader, parent and counselor, etc. Examples of transactions are "the presentation 
of a film, a class discussion, the working of a homework problem, an explanation 
on the margin of a term paper, and the administration of a test" (Stake, 1967). 
According to Stake, transactions are dynamic, whereas antecedents and outcomes 
are relatively static. 

Finally, outcomes involve the impact of instruction and the consequences 
of the program, both immediate and long-range. 

Outcomes, as a body of information, would include measurements 
of the impact of instruction on teachers, administrators, counselors, 
and others. Here too would be data on wear and tear of equipment, 
effects of the learning environment, costs incurred. Outcomes to be 
considered in evaluation include not only those that are evident, or 
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even existent, as learning sessions end; but include applications, 
transfer, and relearning effects which may not be available for 
measurement until long after. (Stake, 1967) 

According to Stake, the evaluator is responsible for preparing a descriptive 
record of what educators intend to happen and of what observers actually find. 
In processing the descriptive data. Stake advises evaluators to look for the 
contingencies among the intended antecedents, transactions and outcomes, 
and to look for the congruence between that which was intended and that which 
is observed. Contingency is measured both logically and empirically. An attempt 
should be made to determine to what degree the intended antecedents, trans- 
actions and outcomes relate logically to one another. Empirical analysis requires 
that data must be collected upon the observed antecedents, transactions and 
outcomes. To determine congruence, one looks at the degree to which the 
intended are actually observed; that is, did what was intended actually happen? 
The evaluator should then describe the amount of congruence for the antecedents, 
transactions and outcomes. 

The judgmental aspects of evaluation can be accomplished with respect to some 
absolute standards of excellence as reflected by personal judgments of experts in 
a given field, or a relative comparison with the characteristics of outcomes 
themselves. The evaluator determines which set of standards to heed and if each 
standard is met. Judging is assigning a weight of importance to each set of 
standards, whether it is absolute or relative. The judgment data are then used 
as input to the educational decision-making process by obtaining an overall 
rating of th? program's merit. 

As to the importance of relative or absolute judgments, other theorists 
have disagreed. Michael Scriven calls for direct comparisons between programs 
in order to find out which program is better for particular purposes. Lee 
Cronbach asks for fewer comparisons and more intense process studies with 
extensive description. It seems that Stake is calling for both complete description 
and judgment incorporating both the absolutes and relatives. One should not 
make a decision with regard to a single characteristic from a single program. 
The evaluator must set the priorities, determine which characteristics he will 
attend to, and decide what type of judgment data he will use. From this process 
a reconimendation is made. 



BASIC REFERENCE on Stake's Model 

Stake's model is described in detail in the following article: 

Stake, Robert E., "The Countenance of Educational Evaluation/' 
Teacher's College Record. 1967, 68, 523-540. 
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B. Stufflebeam's CIPP Model 

Daniel Stufflebeam defines evaluation as a decision-making process which 
involves three major steps: delineating the information to be collected, obtaining 
the information, and providing the information to decision makers. Such 
information, if valid, reliable, timely and credible, should enable decison makers 
to judge the relative value of alternatives with which they are faced. 

Stufflebeam identifies four types of decisions which can be made in 
educational settings: planning, structuring, implementing, and recycling decisions. 
Planning decisions are made to determine objectives; structuring decisions are 
intended to design procedures; implementing decisions are made to utilize and 
refine procedures; and recycling decisions are for purposes of judging and reacting 
to attainments. Corresponding to the four types of educational decisions are 
four kinds of evaluation: context, input, process and product (See Figure 3). 
These are the key components of the CIPP evaluation model. They come at 
different times in the evaluation process, and also serve different decision-making 
functions. 



Figure 3 

Stufflebeam's Four Types of Evaluation 





CONTEXT EVALUATION 


INPUT EVALUATION 


PROCESS EVALUATION 


PRODUCT EVALUATION 


OBJECTIVE 


To define the operating 
context, to identify and 
assess needs and oppor- 
tunities in the context, 
and to diagnose prob- 
lems underlying the 
needs and opportunities. 


To identify and assess system . 
capabifities, available input 
strategies, and designs for 
implementing the strategies. 


To identify of predict, in 
process, defects in the 
procedural design or its 
implementation, to 
provide information for 
the preprogrammed 
decisions, and to maintain 
a record of proceduraf 
events and activities. 


To relate outcome infor- 
mation to objectives and 
to context, input, and 
process information. 


METHOD 


By describing the context, 
by comparing actual and 
intended inputs and out- 
puts, by comparing 
probable and possible 
system performance, and 
analyzing possible 
causes of discrepancies 
between actualities and 
intentions. 


By describing and analyzing 
available human and 
material resources, solution 
strategies, and procedural 
designs for relevance, 
feasibility and economy 
in the course of action to 
be taken. 


By monitoring the activity's 
potential procedural 
barriers and remaining alert 
to unanticipated ones, by 
obtaining specified infor- 
mation for programmed 
dec 'ons, and describing 
the actual process. 


By defining operationally 
and measuring criteria 
associated with the 
objectives, by comparing 
these measurements 
with predetermined 
standards or comparative 
bases, and by interpreting 
the outcomes in terms of 
recorded context, input 
and process information. 


RELATION TO 
DECISION 
MAKING IN 
THE CHANGE 


For deciding upon the 
setting to be served, the 
goafs associated with 
meeting needs or using 
opportunities, and the 
objectives associated with 
solving problems, i.e., for 
pfenning needed changes. 


For selecting sources of 
support, solution 
strategies, and procedural 
designs, i.e., for structur- ■ 
ing change activities. 


For impfementing and 
defining the program design 
and procedure, i.e., for 
effecting process control. 


For deciding to continue, 
terminate, modify, or 
re focus a change activity, 
and for linking the activity 
to other major phases of 
the change process, i.e., 
for recycling change 
activ lies. 
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I ...... 

Context Evaluation comes at the outset of the evaluation, and is primarily 
descriptive in nature. Its purpose is to identify the needs of the environment and 
to define the problems which are to be solved. This type of evaluation also 
involves delineating goals and objectives to be achieved by the program. 

Input Evaluation / Wke Context, is primarily descriptive. Its purpose is to 
provide information rieefessary for selecting a strategy to meet program objectives 
and for developing a plan to implement the strategy. Primary concerns are 
procedures, staffing, and budgeting. 

Once a course,Q;feaction has been selected and implemented. Process 
Evaluation is initiated: Process Evaluation serves to provide information about 
detects in the program during the implementation stages. It also assists the 
project personnel in making decisions in a continual effort to improve the 
program. Data are collected continuously so that the evaluator can help in the 
interpretation of outcomes and also to document when change is needed. 

Product Evaluation determines the effectiveness of the overall project when 
it has completed its full cycle. The purpose of Product Evaluation is to relate 
outcomes to objectives, context, input, and process. Decisions are made at this 
level to continue, terminate, modify or refocus the program. StuffJebeam 
believes that the use of experimental design at this stage is useful, and in many 
cases necessary. 

Common .0 each stage of evaluation is a general structure for implementing 
the evaluation. The structure includes six major parts: focusing the evaluation, 
information collection, information organization, information analysis, infor- 
mation reporting, and the administration of the evaluation. 

Much of what Stuff lebeam says concerning evaluation, particularly 
Process and Product Evaluation, is included in other models and his emphasis 
on description in the early stages of evaluation is similar to Stake's concern 
with descriptive information. The unique feature of Stuff lebeam's model is 
that he combines all of these ideas into a workable, though somewhat cumber- 
some model. 

The positive aspect of this model is that it provides for evaluation at any 
stage in the life of a program. Il» emphasizing continous feedback from the 
evaluator to the decision maker, problems within a program can be identified 
and dealt with as they appear rather than at the conclusion of the program when 
it is too late for corrective action "cO be effective. While Stuff lebeam's model 
can be praised for its comprehensiveness, i ^al-:life evaluations rarely are involved 
with planning and structuring decisions. T' e procedures he suggests are both 
complex and costly and would require a \cir}e staff to implement satisfactorily. 
And while providing useful information oii: incision makers is stressed at every 
level of evaluation, the steps or processes involved in making the decisions are 
not clearly defined. 



BASIC REFERENCE on Stuff lebeam's Model 



Stufflebeam's m^jor book. on evaluation has been cited in an 
earlier Basic Reference. A paper which summarizes the main 
aspects of his model was presented at the:- 

Eleventh Annual Phi Delta Kappa Symposium on Educational 
Research, The Ohio State University, June 24, 1970. 
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C. Provus' Discrepancy. Model 

Evaluation as defined by Malcom Provus encompasses three processes: 
agreement upon program standards, program assessment, and program 
improvement. The model is intended to facilitate design changes and <iata 
gathering essential to making judgments about the effectiveness of a program. 
The information collected may be used to determine whether to maintain or 
terminate a program at any stage of its development. 

The model is intended to guide the evaluator in making comparisons 
between a program and its design on one hand and a series of agreed-upon 
program standards on the other. The evaluation staff collects information 
essential to program improvement and notes discrepancies between performance 
and standards. Every question involves a criterion (C), new information (I), and 
a decision (D). Evaluation provides the new information. Every aspect of the 
program is evaluated and comparisons are made of actual events with expected 
events at various stages of the program. Using a "feedback loop" approach, the 
evaluator provides discrepancy information that is useful to program managers 
in changing the program performance or the design standard so that the two 
are equalized. 

To help explain the model, a diagram is presented depicting discrepancy 
evaluation (See Figure 4). The model consists of five stages, each providing a 
comparison of performance and standard; discrepancy information is then fed 
back for program adjustments. In Stage 1 (Design) a definition of program 
standards and structure is set by the program staff. (The content contained 
across all stages is the program structure: Inputs x Process = Outputs.) Infor- 
mation is collected on program specifications and a comparison is made 
between the program design and a set of design criteria which delineate the 
structural and theoretical base of the program as defined by an outside 
consultant. If a discrepancy exists, the information is fed back into the pro- 
gram so that a change can be made. 

Figure 4 

Diagram of Provus' Discrepancy Model 
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DacisiorHVtaking Prooats 

Discrepancy Information: 



1 , Go on to next stage 

2, Recycle the stage after a change has been made 

3. Recycle to the first stage after a change has been made 

4. Terminate the project 



In Stage 2 (Installation) the reality of the program is compared with the 
program design arrived at in Stage 1 . Is the installation of the program the 
program design? If not, discrepancy information is used for change, either in 
Stage 2-performance change-or back to Stage 1 for a standard change. 

In Stage 3 (Process) the relationship between process and interim products 
is established. The hypothesized activities which go on between inputs and the 
program outputs are compared with actual activities (interim products.) To 
judge whether the program is achieving its enabling objectives, discrepancy 
information based on the actual program performance of students is analyzed 
and any necessary corrective alternatives are examined and installed. Thus, the 
third stage contributes to the shaping of a successful program. 

Stage 4 (Product) compares actual terminal products with hypothesized 
ones. The outcomes of the program are assessed and the program is tested for 
its generalizability. Finally, Stage 5 (Product Comparison) assists in choosing 
between programs by way of cost-benefit analysis to determine program 
efficiency. 

The information collected at each stage and the accompanying criteria 
are as follows: 

STAGES INFORMATION COLLECTED 

Design Program Structure: l(P)-0 

Installation Input-Process Performance 

Process Process-Output Performance 



CRITERION 

Design Criteria 

Program Design v^,. 

Process-Product 
Relationships 

Terminal Objectives 

Cost Analysis 



Product Input-Output Relationships 

Product Comparison Input-Output Comparability 

The model is a complex one with an intricate system of feedback loops. 
Provus' first four stages are concerned with single program evalucition and the 
development of that program. Stage 4 seems to be right out of a behavioral 
objectives reference manual— the evaluation of terminal objectives in reference 
to corresponding inputs. The whole idea of comparison with a standard reminds 
one of Stake's idea of Congruence and making judgments with absolute, 
standards given by experts. Provus also seems to borrow from Scriven in that 
formative evaluation is used in the first four stages while summative evaluation 
is used in the fifth stage in the comparisons of similar programs. 



BASIC REFERENCE on Provus' Modal 



Provus' book. Discrepancy Evaluation (McCutchan Publishing Co., 
Berkeley, California, (1972), is an extremely complicated treatment 
of his model, perhaps an "over-elaboration" on the basic idea. 
Therefore, I would suggest that you further pursue this model 
through the following reference: 

Provus, Malcom. "Evaluation of Ongoing Programs in the 
Public School Systems." In R.W. Tyler (Ed.), Educational 
Evaluation: New Roles, New Means. The Sixty-Eighth Year- 
book of the National Society for the Study of Education, 
Part !!. Chicago: NSSE, 1969. 
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D. Eash's Differential Evaluation Model 

The model developed by Maurice J. Eash was designed specifically for the 
evaluation of new and innovative programs. Since programs for the gifted and talented 
are often characterized by flexibility and innovation, Eash feels that new programs 
need the freedom to evolve and clarify objectives as experience dictates. For this 
reason, he formulated a three stage evaluation methodology that parallels the stages 
of program maturation. Evaluation is carried out along a continuum, with each 
stage of a program being evaluated in terms of its particular activities and goals. 

The continuum that Eash suggests is composed of three models: the initiatory 
model, the development model and the integrated model (See Figure 5). The 
major processes of the initiatory model consist of the planning of goals, specifications 
and operations. In the initiatory model there are few outcomes from the program 
to study. The evaluator analyzes the functions of various committees and makes 
recommendations for their future work. The developmental model is characterized 
by the actual construction and testing of a program in field operation. Implementaion 
of the developmental model takes place after program activities begin to stabilize. 
In the integrated model the program is clear on its goals, can predict with reasonable 
accuracy the outcomes of program activities, and can generate evaluation data 
for internal adjustments. 

The dimensions Eash identifies for consideration in the evaluation process 
for each program model are: effort (how time is spent), effect (products and 
outcomes), and efficiency (the relationship of the efforts and resources to the 
effects achieved). The application of these factors will be specific to the nature 
of each model and will therefore seek different data. Figure 6 contains specific 
questions which the evaluator would raise in each model and for each dimension. 
Note that as the program moves from the .initiatory to integrated model, increased 
emphasis is placed on product evaluation , i.e., effect. 

Perhaps the most valuable feature of Eash's model is its allowance for 
modifications in program objectives over time. It makes sense— both in theory 
and practice— to differentiate evaluation procedures for the different stages in 
program development. It appears, though, that this plan for evaluation is less 
specific to gifted education than its title leads the reader to believe. Except for 
using gifted programs as the examples in the tables, this model is equally applicable 
for the evaluation of any new or innovative program— be it in special education or * 
general education. Thus, rather than a specific method for the evaluation of.gifted 
programs, Eash's contribution can be seen as one of more general relevance to 
evaluation methodology. 



BASIC REFERENCE on Eash's Model 



Eash's model has been described in its entirety in the following 
paper. Copies of the paper may be obtained from the ERIC Clearing- 
house on the Handicapped and Gifted or by writing to Dr. Eash 
at the University of Illinois, Chicago Circle Campus, Chicago, 
Illinois. 

Eash, Maurice J. "Issues in Evaluation and Accountability in 
Special Programs for Gifted and Talented Children." 
Unpublished Paper, 1971. 



35 



Figure 5 

Three Levels of Program Models in Special 
Programs for Gifted and Talented 



MODELS 



DEVELOPMENTAL MODELS 



INTEGRATED 



INITIATORY 

Models are vague, intuitive in effects 
to be achieved. Objectives are stated 
as general outconnes and social goods 
to be achieved. There is nnuch concern 
with theory. The debates on alterna- 
tives are theoretical rather than opera- 
tional or data based. Justification of 
the program may be drawn from analo- 
gous programs in other contexts or be 
based on philosophical assumptions. 
Details for operationatizing the pro- 
posal are sketchy. 



Precis of a Program 

A special program for gifted and 
talented children is drawn up. 
Decisions on the form it will take; 
special classes, enrichment, indepen- 
dant tutorials for the mix of these are 
still open. There is tack of agreement 
on definition of clients. Who is a 
gifted or talented student? How 
should he be educated? Should he be 
identified? At what grade? By whom? 
Will there be extra monies allocated 
to the education of these students? 
Will there be a need to establish a 
separate administrative unit for this 
program? What type of research wilt 
be conducted on a program? When 
will parents be involved? A com- 
mittee has been set up to resolve 
some of these issues. Administrative 
responsibilities and a sum of money 
for planning have been allocated. The 
committee has been meeting for one 
year, a set of minutes, a list of con- 
sultants and a description of the field 
trips to visit programs for gifted 
children exists. 



Models, where a mixture of objectives 
prevails; macro objectives give general 
guidance and some micro objectives are 
defined. Objectives still seem to be shift- 
ing and the model still takes different • 
forms in individual staff descriptions. 
There is more concern with operational 
alternatives than a given alternative. While 
the program is operating there are many 
unknowns and frequently considerable 
improvisation. 



Precis of a Program 

One special program for gifted and 
talented children has been underway two 
years. Fifty children are involved. In 
some cases teachers nominate students 
for the program, in others they are 
selected on l3asis of test scores, the first 
year students spend four hours per week 
in the program, the second year this has 
been extended to six. The program has 
focused on scientific interests though 
there is concern about including more 
humanities. One teacher made arrange- 
ments for 25 of the students to see the 
Old Vic perform at the local college. 
Some data, mostly of a descriptive 
nature has been collected on the 
students/ their achievements and the 
program. Teachers do not have fixed 
style for instruction, the instruction 
reflects personal teaching style. 



MODELS 

Models have specific objectives to 
be achieved. There is monitoring of 
operation. Relationships of treatment 
(what is done educally) and effects 
(outcomes) are specified, and repro- 
ducibility is enhanced by elaborated 
descriptions of the model in operation. 
Logical relationships are explicated, 
and empirical data are being collected. 
The outcomes are being assessed and 
the range of effects are capable of 
being attributed to the program treat- 
ment. 



Precis of a Program 

A program for gifted and talented 
students has been in operation for five 
years. Open-ended instruction is 
featured with teachers and students co- 
operatively planning the curriculum 
for three months at a time. The Director 
of Research for the school district 
monitors the program through teachers' 
records, student interviews and regular 
classroom visitations. Program outcomes 
are investigated through their effect on 
student's achievement and interest. A 
contrast group of students, not in a 
special program, in a neighboring school 
district with a similar student body is 
supplying comparable data on special 
programs' influence on the regular 
program. At the end of the five years 
a summer workshop composed of 
teachers and pupils in the program in 
conjunction with administrators and 
university consultants will draw up the 
program description for the next three 
years. Decisions will be rendered on the 
program organization, the selection and 
retention of students and the research 
to be conducted. 
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Figure 6 

Differential Evaluation in Three Program Models 
INITATORY MODEL DEVELOPMENTAL MODEL 



INTEGRATED MODEL 



1 . What have been the main 
directions of the committee's 
efforts? 

2. What has been the level of partici- 
pation among the committee 
members? 

EFFORT 3. Has the committee broadened its 
constituency and recognized the 
socio political aspecu of its efforts? 

4. How much time has been spent on 
certain phases of the program? 



1 . What have been the main thrusts of 
the program's efforts? 

2. What objectives have received the 
major attention? 



3. 



EFFECT 



5. 



Who has been involved in the program, 
to what extent, voluntary or mandated, 
volunteer or paid? 

Where has the support for the program 
emanated; what has been the total 
developmental costs— financial and 
psychic? 

How much total time has been spent? 
What parts of the program are consum- 
ing the bulk of time? 



1 . What are the major goals the 
program is trying to attain? 
Who is involved in the effort? 

2. What percentage of staff and 
student time is committed to 
the program? Total time? 

3. What data are available that 
permit building a history of 
the effort and projecting a 
scenario for future thrusts? 

4. What areas of effort are per- 
ceived as worth while by the 
different role participants? 



What is the level of knowledge 
manifested in the committee on 
special programs for the gifted and 
talented? 



2. Are the committee members 
conversant with issues, trends 
and programs? 

3. What is the present stage of the 
plans, are they near operational- 
izing? 

4. What are the main impediments to 
formulating a developmental model 
program? 



3. 



What data on functioning of the pro- 
gram have been collected or can be 
collected? 



What have the effects been on program 
students, other students, teachers, 
parents and administrators? 

Have the data on effects been used to 
modify or shape the program, explore 
alternatives? 



4. Can the effects on students be attrib- 
utable to the program? 

5. Have there been any unanticipated 
effects? 



1 . What are the programs short- 
range effects on students in 
the program, students not in 
the program, teachers, parents 
and administrators? Are data 
available to study both process 
and product effects? 

2. Is any provision made for 
studying long range effects? 

3. Can the desired effects 
si'^lci^ ip the original goals 

be attributed to the program? 

4. Have there been any unanti- 
cipated effects? 



EFFICIENCY 2. 



Ooes the committee have an 
organized plan for carrying out its 
work, with deadlines and completion 
schedules for phases of activities? 

Is the committee clear on its 
responsibility to the Board of Edu- 
cation and superintendent? 

Given the amount of time and 
money invested has a useful 
product emerged? How far are 
they from an operating program? 



1 . Are there records or other evidence 
that program problems are being 
systematically encountered and 
resolved? 

2. How does the cost on this program 
compare with costs on other programs 
in the district and in other districts? 

3. What goals seem within attainment? 
What goals have not been attained? 



4. Given the program's experience, what 
will be the approximate cost of an 
integrated program model? 



1 . Ar* problems systematically 
studied? Are the participants 
conversant with the decision 
making process? Has it been 
scrutinized? 

2. What is the cost of this program 
compared with other programs in 
the district and similar programs? 

3. How do these costs project out 
for the future now that develop- 
mental costs are largely met? 

4. What has been the cost of attain- 
ing certain effects, what trade- 
offs were made in the interest 

of cost? 
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E. Renzulli and Ward's "DESDEG" Model 

The Diagnostic and Evaluative Scales for Differential Education for the 
Gifted (DESDEG) were developed as a guide for both self-study and for 
assessment by an external evaluation team. The model represents an attempt 
to bridge the gap between theory and practice in the field of evaluation as 
it applies to programs for the gifted. Although DESDEG was designed mainly 
for purposes of program evaluation, the authors suggest that it can also be 
used as a guide for program planning and development. 

DESDEG consists of five interrelated components. Part \, the Manual, 
provides a theoretical rationale underlying this particular approach to evaluation 
and an explanation of the steps involved in carrying out a comprehensive 
evaluative study. Part 1 1, the Evaluative Scales, consists of a set of fifteen 
"Program Requirements" (PRs) that were judged by a group of experts to 
be important characteristics of comprehensive programs for the gifted. The 
PRs are organized around five "Key Features" which represent general areas 
of consideration in program development and implementation (See Figure 7). 
The PRs may be viewed as guides that provide structure and focus to the 



Figure? 

Overview of the DESDEG Model 

Key Feature A : Philosophy and Objectives 

Program Requirement 1 : Existence and Adequacy of a Document 
Program Requirement 2: Application of the Document 

Key Feature B: Student Identificatiori and Placement 

Program Requirement 3: Validity of Conception and Adequacy 

of Procedures 

Program Requirement 4: Appropriateness of Relationship Between 

Capacity and Curriculum 

Key Feature C: The Curriculum 

Program Requirement 5: Relevance of Conception 

Comprehensiveness 
Articulation 

Adequacy of Instructional Facilities 



Program Requirement 6: 
Program Requirement 7: 
Program Requirement 8: 

Key Feature D: The Teacher 

Program Requirement 9: 
Program Requirement 10: 



Selection 
Training 



Key Feature £: Program Organization and Operation 



Program Requirement 1 1 
Program Requirement 12 
Program Requirement 13 
Program Requirement 14 
Program Requirement 15 



General Staff Orientation 

Administrative Responsibility and Leadership 

Functional Adequacy of the Organization 

Financial Allocation 

Provision for Evaluation 
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observations, the inquiries and the interpretations of the evaluator, after he has 
analyzed and checked the data available in the Basic Information Forms (see 
description below). The specificity of each of these requirements and their 
deliberately ordered parallelism and comprehensiveness makes the diagnostic 
potential of the instrument especially valuable in suggesting changes and 
making recommendations relating to particular program practices. 

The program requirements may be thought of as statements of certain 
principles about education for the gifted that are found in the literature, 
and which depict ideally-conceived educational practices for exceptionally 
able students. They do not pertain exclusively to any given pattern of 
program organization, but rather attempt to embrace excellent practices 
whatever the nature of the administrative structure of the program; practices 
that can and should be inaugurated in view of the behavioral potential of 
superior students. 

Each program requirement sierves as a focal idea around which a set of 
five Scale Standards has been developed. Thus, the Scale Standards depict 
practices or provisions of varying merit that are derivatives of the respective 
Program Requirements. 

Part I II of DESDEG consists of The Basic Information Forms. These 
forms are intended to provide the evaluator with a comprehensive inventory 
of factual information about all aspects of a program that might have some 
bearing on the final judgments and recommendations which he must make. 
It should be emphasized that the information sought through this component 
of the DESDEG plan is purely descriptive; therefore, an attempt has been 
made to structure the Basic Information Forms in a way that allows for a 
relatively objective collection of data, The forms have been designed to be 
completed by persons within the program rather than by external evaluators; 
and for this reason, the information has been sought in a manner that requires 
a minimum amount of interpretation on the part of persons providing the data. 
The solicitation of information from knowledgeable program participants is 
an economical means for gathering data; and yet, the non-judgmental nature 
of the information requested helps to avoid the unconscious bias that well- 
intentioned participants often bring to an evaluative situation. Thus, a 
distinction is maintained between the descriptive and judgmental functions; 
description being the responsibility of persons within the program, while the 
interpretation of descriptive information leading to judgments remains the 
responsibility of the professional evaluator. The Basic Information Forms 
are organized and keyed according to their relationship to each of the 
five key features. 

Part IV of DESDEG is called The Evaluator's Workbook. This section is 
intended to assist the evaluator in processing information derived from the 
Basic Information Forms and from observations that are made while visiting 
the program. Thus, it may be thought of as a practical aid for translating 
descriptive data and observed events into judgments. The Workbook follows 
the same pattern of organization as the two previous components (i.e.. Key 
Features and Program Requirements), and consists mainly of suggestions to 
the evaluator about important factors fo be considered in arriving at final 
judgments for each of the programs' requirements. Illustrations of transformations 
from descriptive data to scale standards are provided, modes of inquiry are 
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suggested, and examples are given of extenuating circumstances that may lead 
to special consideration in assigning or withholding the designation of scale 
values in certain areas of the program. Space is provided for recording the 
results of interviews and observations, for summarizing information derived 
from various sources, and for listing questions and points that are in need 
of further clarification. 

The final section of DESDEG is The Summary Report This report is 
again organized in a pattern consistent with the preceding components so 
as to allow the evaluative findings and recommendations to be made in 
relation to the respective key features and program requirements. All ; 
numerical ratings from the Scale Standards are transferred to statistical 
and graphic summary forms. 

Although the quantitative aspects of the Summary Report will no 
doubt be of interest to school personnel, the most important function of 
this component will be served by the nariative summary and recommendations 
that the evaluator is asked to make for each of the Program Requirements. 
This section should contain (a) a brief statement telling why the rating on 
a given Program Requirement was earned, and (b) specific suggestions for 
action that is needed to bring about an improvement in the Program Requirement 
Needless to say, such suggestions must be based on local circumstances and 
resources; and should be guided by the total conglomerate of information 
which the evaluator has processed. Provisions also are made in the narrative 
sections of the Summary Report for pointing out the strongest features of 
the program being evaluated. 




10 



BASIC REFERENCE on the DESDEG Model 



The entire DESDEG plan for progrann evaluation, including 
each of the five connponents described above can be found in: 

Renzulli, Joseph S. and Ward, Virgil S. Diagnostic and 
Evaluative Scales for Differential Education for the Gifted. 

Storrs: University of Connecticut, 1969. 



II. Basic Concepts in Program Evaluation 

As the literature on evaluation grows in size and connplexity, a whole new 
language of evaluation is ennerging that describes the concepts which have 
helped to create this area of specialization. Although nnany of the concepts 
have been adopted o; adapted fronn other areas such as psychonnetrics, 
research design, and instructional technology, a snnall nunnber of basic 
concepts have beconne almost essential to understanding the language of 
the evaluator. Almost all professional evaluators are familiar with these 
concepts, and although they may disagree on some of the subtle ramifications 
of their meanings, the basic concepts provide a useful vehicle for facilitating 
communication in the design and implementation of evaluative studies. 



A 



I KNOW mr YOU' BEUEVE XOU UNDERSTAND 
WHAT YOU TWINK I SMO, BUT I AM NflfT SURE 
XCO REAIUE TWAT WHAT VOO HEAeO IS WUAT I MEANt 



The purpose of this section is to review some of the basic concepts that 
help to define the field of program evaluation. The would-be evaluator and 
consultant who is rendering technical assistance wilt no doubt want to explore 
these concepts in much greater depth, and for this reason some carefully 
selected Basic References have been included. The section should provide the 
person who will seek the services of a professional evaluator with enough 
information to communicate meaningfully with those persons who will carry 
out an evaluation study. Space limitations.and the general purpose of this 
Guidebook preclude an exhaustive treatment of the basic concepts; and 
therefore, this section will attempt to focus upon general problems and issues 
related to each concept and the ways in which these problems and issues have 
special relevance to the task of evaluating programs for the gifted. 



The first two concepts discussed below, Formative and Sunnmative 
Evaluations, nnay be thought of as evaluation designs. They represent the 
predeternnined plans that guide the ways in which an evaluation will be 
carried out and the role that evaluation will fulfill in the overall operation 
of a project. It should be ennphasized that an evaluation need not be either 
fornnative or sunnnnative, but rather can be a connbination of both designs. 

The other three concepts discussed in this section, Product, Process, and 
Presage Evaluations, should be thought of as types of evaluative data — that is, 
the kind of infornriotion that an evaluator focuses upon in organizing and 
conducting an evaluative study. (This nnay be sonnewhat of an oversinnplification 
of Product, Process, and Presage; but thinking of these three concepts as types 
of data will help to eliminate some of the confusion that invariably arises when 
discussing evaluation designs.) Decisions regarding which types of data an 
evaluator will seek are, of course, also based on the role that evaluation is 
expected to play; and thus, we begin to see relationships forming between 
Formative and Summative Evaluations on one hand and Product, Process, and 
Presage on the other. The sections that follow will attempt to point out these 
relationships, but suffice it to say at this time that most field evaluations take 
into account several combinations of the concepts discussed below. 

. A. Formative Evaluation 

A great deal has been written on formative evaluation and although much 
of the writing is an overelaboration on an essentially simple concept, this 
concept is emerging as one of the most powerful ideas in present-day thinking 
about evaluations. Michael Scriven, the person who originated the concept, 
defines formative evaluation as "simply outcome evaluation at an intermediate 
stage in the development of [whatever it is that you are evaluating] " (Scriven, 
1967, p. 51 ). The role of formative evaluation, according to Scriven, "is to 
discover deficiencies and successes in the intermediate versions" of educational 
programs and activities (p. 51 ). We note here that the emphasis is on when 
the data are gathered (intermediate stages as opposed to end-of -program data) 
rather than on the types of data that are being used. The major purpose of 
formative evaluation is to provide continuous in-process feedback so that 
appropriate modifications and revisions can be made in a program as the 
program develops and matures. One of the primary advantages of formative 
evaluation is that the data are gathered in close proximity to specific components 
of a program, and thus it has greater potential for pinpointing the successes 
and failures of particular program activities. 

All types of data (Product, Process, Presage) can be used in formative 
evaluation but it is important to keep a few basic rules in mind. First, a 
systematic feedback mechanism must be developed so that information reaches 
decision makers in time to institute changes that are deemed necessary. Secondly, 
decision makers at each level of decision-making responsibility (see Chapter One, 
Part I l-A) must make a sincere commitment to change.* Flexibility is a key word 

*l recently was the evaluator of a supposedly innovative program in nursing education. The 
program director seemed committed to the formative evaluation concept, but the first time 
negative findings were reporte'd she perceived the information as being a threat to her "grand 
design" for a nursing curriculum. It became clear that the director viewed the role of evaluation 
as one of program verification rather than program improvement. (Needless to say, we parted 
company at this point!) 
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in formative evaluation and unless program developers are willing to make 
both major and minor modifications, there is little value in providing them 
with in-process feedback. The third and perhaps most important rule to 
follow in formative evaluation is that information should be collected on 
identifiable program activities that we can do something about. Formative 
evaluation data are useless unless they tell us where to make changes and 
the changes are within the realm of possibility. This is an important point 
to keep in mind when selecting instruments to measure student growth, A 
standardized achievement test may tell us if students have made gains (or 
failed to make gains) in a particular area such as language arts, but it will 
not fulfill the role of formative evaluation if we cannot relate the scores to 
certain types of instructional materials, teaching strategies, classroom climates, 
or an interaction among various combinations of these variables. This is 
especially true when we are dealing with the effectiveness of a particular 
body of curricular materials. Fortunately, many of the "big" curriculum 
projects {Man: A Course of Study, Science - A Process Approach, etc.) 
have been carefully evaluated in their developmental stages and the eyaluator 
should not be expected to fulfill the role of the research and development 
person. Comprehensive curriculum evaluation requires a tremendous commitment 
in terms of personnel and money; and for this reason, it may very well be a 
luxury that can only be carried out in an ideal fashion by well-funded research 
and development projects. If, however, the evaluator is expected to engage 
in formative curriculum evaluation, he or she can benefit from some of the 
strategies that have grown out of the experimental curriculum projects. 
These projects have generally relied on carefully developed mastery tests 
which bear direct relationships to specific areas of the curriculum. The Basic 
Reference which follows is an excellent source for persons who need to 
develop skills in the construction of such tests. The recommended strategies 
are based on a detailed analysis of content areas and process objectives which 
are classified according to the Taxonomies of Educational Objectives. 
Instructions are given for drawing up tables of specifications and constructing 
formative evaluation tests. 



BASIC REFERENCE on Curriculum Evaluation 



This large volume (more than 900 pages) is a handbook in the 
truest sense of the word and is not intended to be read from 
cover to cover. Evaluation techniques are presented in a how- 
to-do-it fashion for each of the major subject fields and levels 
of education. 

Bloom, Benjamin S., J. Thomas Hastings, and George F. • 
Madaus. Handbook of Formative and Summative Evaluation 
of Studen t Learning. 

New York: McGraw-Hill Book Company, 1971 . 
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At this point it would probably be wise to reemphasize the fact that 
curriculum evaluation based on mastery testing represents only one dimension 
of total program evaluation. Persons who are engaged in formative evaluation 
will no doubt want to provide continuous feedback in other areas where 
corrective action can be taken such as reactions of parents toward the program, 
the effectiveness of in-service training, etc. In the next chapter we will discuss 
methods for identifying those aspects of a program which should be evaluated, 
and at that point the evaluator should decide which aspects are best approached 
through the formative evaluation design. 




B. Summative Evaluation 

Summative evaluation differs from formative evaluation mainly in the 
role that it fulfills. Whereas formative evaluation is directed toward program 
revision and improvement through continuous feedback, summative evaluation 
is more concerned with overall program effectiveness. Thus, summative 
information is more likely to be used in making decisions about program 
adoption or continuation. While the results of summative evaluation are no 
doubt of interest to persons who develop and operate programs, they are 
primarily gathered for those persons who will decide the fate of programs. 
This is an important point to keep in mind. Formative evaluation fulfills a 
very noble educational role and has emerged as the "fair-haired boy" of 
evaluation designs; but summative evaluation plays an equally important 
political role. (I use the word, "political" here in its ancient and honorable 
context, which refers to policy making.) Thus, boards of education or 
funding agencies may be more interested in summative information. 
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Although summative data might be gathered throughout the course of 
a program, the "pure" summative evaluator avoids giving any feedback 
until the end of the program because he wants to see how the program works 
in its natural (unaltered) form. In this respect, summative evaluation resembles 
the classic approach to experimental research design; i.e., holding the independent 
variable (program) constant in order to discover what changes it produces. The 
main difference between summative evaluation and experimental research 
design is that the researcher is usually comparing alternative treatments under 
highly controlled conditions. These conditions almost always include the 
random assignment of students to experimental treatment groups and to 
control groups. Although most evaluators would like to respect as many of 
the mandates of good research as possible, this is seldom feasible in field 
evaluations. As has been indicated earlier, programs for the gifted^aire'::often 
characterized by a great deal of variety and individualization, and it would be 
difficult to do high quality research without very substantial resources. The 
summative evaluator can, however, use the same instruments as the formative 
evaluator in documenting the overall effects of a program. But in this case 
there is a greater need to show growth over relatively long periods of time, and 
for this reason it may be necessary to gather data at the beginning and end 
of a program. 

Up to this point we have been discussing formative and summative 
evaluation as if they are two separate and distinct entities. In actual practice, 
the evaluator will most often use a combination of approaches and some of 
his instruments can serve the dual purpose of providing in-process feedback 
for practitioners anc/ documenting student growth for decision-making bodies. 
The best guide for determining what types of data will be most useful for 
practitioners and decision makers is to survey each group at the outset of an 
evaluation study. The techniques for doing this are discussed in the next 
chapter under the heading of Front-End Analysis. 

C. Product or "Pay-off" Evaluation 

Since educational programs are intended to produce certain changes in 
the attributes of students, product evaluation can be thought of as the 
assessment of observable and measurable student outcomes that result from 
a particular educational endeavor. Assuming that there is some consensus 
about the desirability of intended outcomes, these outcomes then become 
the "pure pay-off" of an educational program. The only important evaluative 
data, according to the hard-nosed pay-off evaluator, are documented indications 
of change in student performance, change that would not have taken place 
had the student not been enrolled in a particular course or taken part in a 
certain educational activity. Scriven (1967) has pointed out the many 
problems that are associated with pay-off evaluation, not the least of which 
is the difficulty of determining how we will judge evaluative data once they 

have been gathered. According to Scriven: 

♦* 

The evaluator will have to relate the students' performance 
to some abstract criterion, whether it is his [the evaluator's] 
conception of an adequate professional comprehension, or 
what he thinks it is reasonable to expect [a student] to 
understand. . . 
[Scriven, 1967, pp. 60-61] 
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In other words, merely obtaining objective measures or descriptions of 
student growth does not enable the evaluator to make qualitative judgments 
about what has been learned (e.g., How flfooc/ is an increase of 10 points in ' 
the mean score for a particular group?). And while Scriyen's concern has 
been expressed mainly in terms of comprehension, we can readily see that 
the problem of establishing abstract criteria for judging educational products 
will be compounded as we focus our attention on higher level processes such 
as creativity and problem solving abilities. In the final analysis, some form of 
human judgment must be brought to bear on objective findings; and thus, one 
of the major responsibilities of the product evaluator is to determine what 
types of information are most necessary for facilitating the judgment process. 

The most obvious and popular type of product evaluation data has been 
scores on standardized and teacher-made tests. Some of the problems involved 
in using tests to assess the quality of programs for the gifted and talented have 
been discussed in Chapter One, Part Ill.Scriven (1967) further elaborates on 
these ideas by pointing out that "the performance of students on the final 
tests, as upon the tests at intermediate stages, must be analyzed in order to 
determine the exact location of shortcomings of comprehension, shortages 
of essential facts, lack of practice in basic skills, etc." (Scriven, 1967, p. 61). 
In other words, test scores, in and of themselves, tell us nothing about cause 
^ and effect relationships and the only way that we will be able to pinpoint 
such relationships is through a thorough analysis of test items as they relate 
to course content. 

In recent years the development of a different concept in testing, referred 
to as criterion-referenced measurement, shows promise of making more 
effective use of tests in program evaluation. Whereas traditional norm- 
referenced tests yield only scores that show an individual's relative standing 
in comparison to a norm group, the newer criterion-referenced tests are 
designed to show a student's accomplishments in particular areas in relation 
to a level of performance that he or she will be expected to achieve. For 
example, rather than saying that a student is "above average" or "scoring 
at the 75th percentile" (a normative statement), criterion-referenced tests 
offer statements such as: "The student correctly spelled 18 our of 20 words 
from the basic fourth grade vocabulary list in the Second Unit of instruction." 
This statement tells us in a relatively precise fashion which skills a student 
has mastered. From statements such as these we can begin to make inferences 
about how effectively particular segments of instruction are carried out and 
where specific modifications should be introduced into a program. Thus, 
criterion-referenced testing can help us to accomplish the basic task of 
evaluation which is program modification through systematic feedback. 
At the same time, criterion-referenced tests can help us to serve individual 
students better by determining exactly where remedial instruction may be 
necessary and/or whether or not a student is ready to go on to the next 
step in a learning hierarchy. 

As far as product evaluation is concerned, it sounds as though criterion- 
referenced testing may be the answer to our prayers; but unfortunately, 
such is not the case. Although this type of measurement \s a potentially 
valuable means for assessing educational products and identifying specific 
strengths and weaknesses in instructional programs, a very important caution 
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must be pointed out. Up to this point in time, the criterion-referenced 
testing movement has been closely tied to the behavioral objectives movement; 
and as such, there has been a tendency to concentrate mainly on basic 
skills and limited types of learning activities. We have already discussed the 
high-level-objectives issue, but we should reemphasize in this context (i.e., 
product evaluation) that unless criterion-referenced tests truly assess the 
types of learning that are appropriate for gifted and talented students, their 
use may have the same limiting effects on programs as the rigid application 
of norm-referenced tests. The usefulness of the criterion-referenced measurement 
concept represents a significant advance in evaluation methodology, but we 
must recognize that it is the responsibility of field evaluators Xo apply this 
concept appropriately. This can only be accomplished through the careful 
selection and/or development of instruments that validly assess performance 
at the higher levels of learning. 



BASIC REFERENCE on Criterion-Referenced Measurement 



Articles on criterion-referenced testing are popping up every- 
where but many of them deal only with guidelines for developing 
tests of this type. I have selected the following article for our 
Basic Reference in this area because it covers the topic compre- 
hensively and seems to give the greatest amount of direction . 
toward applying this concept to higher levels of-^leafning. The 
article is at times very technical, but it deals with many of the 
complexities of the issue with which the evaluator should be 
aware. 

Millman, Jason. Chapter 6: "Criterion-Referenced Measurement." 
In W. James Popham (Ed.), Evaluation in Education: 
Current Applications. 

Berkeley, California: McCutchan Publishing Corporation, 
1974, pp. 311-397. 



Up to this point we have discussed product evaluation mainly in terms 
of information yielded by tests. Although this is typically the case, there 
are other alternatives for gathering and judging product information. Ratings 
of student products (or performance) by experts is one way in which the 
quality of creative work can be assessed. For example, in a program for 
gifted and talented students in Warwick, Rhode Island, a group of artists 
and writers were asked to rate students' work that was completed at various 
stages throughout the program. A copy of the rating form which was developed 
for this purpose is included in Appendix B. 
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ALL I DID WA^ CHtP AWAV ANYTHING THAT 
D\DNT LOOK LIKE AN ELtPMANt" 



If specific student attitudes are listed as objectives of a program, the 
measurement of such attitudes can also be considered product data. Numerous 
instruments for the measurement of all types of attitudes have been collected 
in a variety of sourcebooks which are listed in the bibliography of this Guidebook, 
Sample instruments for the measurement of both student and parent attitudes 
are included in the appendices. 

Another type of product data falls under the classification of ''frequency 
counts." This type of data is typically gathered through the use of logs, 
checklists or an analysis of school, records. Once again, frequency counts 
qualify as product data if they reflect the accomplishment of an important 
program objective. Thus, for example, if one of the stated objectives of a 
particular program is: "To increase by 50 percent the number of science books 
that students select for independent reading/' we can evaluate this objective 
by some relatively simple record keeping. We should keep in mind, however, 
that this is a very different kind of product from the qualitative assessment 
of student performance. 

Although product data (especially in the form of test scores) are usually 
used in summative evaluation studies, they can also be used in formative 
evaluation. As has been indicated earlier, the purpose (role), for which data 
are gathered and the frequency with which information is fed back into the 
program are the main factors which distinguish between formative and 
summative evaluations. It is important to reemphasize, however, that when 
product data are used for formative purposes, it must show cause/effect 
relationships and give specific direction to corrective actions that are designed 
to bring about program improvement. 
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It might be worthwhile to end this section by pointing out that there 
is still a fair amount of confusion among evaluators about the similarities 
and differences between good product evaluation and research. The techniques 
and instruments used by the evaluator and researcher overlap considerably 
and the difference can only be expressed by defining the purpose of a particular 
study. Theoretically, the researcher's efforts should be directed toward 
discovering cause/effect relationships and predicting. to what degree these 
relationships can be generalized to other (similar) populations and replicated 
under similar circumstances. The evaluator, on the other hand, has the 
responsibility of finding out how well a particular program operates, given 
certain conditionsvjjn reality, however, the evaluator is implicitly making 
predictions when he recommends continuation (or discontinuation) of a 
program or^par%i€ular practices. 

D. Process Evaluation 

I n view of the many problems that product evaluation presents in assessing 
for the gifted and talented, there is clearly a need to seek additional kinds of 
data which show promise of determining the effectiveness of particular program 
activities. Process evaluation is concerned with assessing those aspects of 
student and teacher behavior which are considered to be worthwhile in their 
own right. These behaviors or processes are the teaching strategies and learning 
activities that are believed to be necessary in order to bring about desired 
educational products. In other words, process evaluation is concerned with 
"what-goes-on" in a learning situation rather than "what-comes-out-of-it." 

Although there is some disagreement among educators about what 
constitutes a process, most evaluators agree that assessing the actual dynamics 
of a learning activity can provide very valuable insights about the strengths 
and weaknesses of certain educational practices. The assessment of educational 
processes is almost always used in formative evaluation studies. Process data 
are usually gathered to give immediate feedback to teachers so that they can 
make desired modifications in their instructional practices. Process data can 
be used in summative evaluation reports but a great deal of caution must be 
exercised when it is used for summative purposes. Teachers may be genuinely 
interested in "taking-a-look-at-themselves" so that they can improve their 
teaching techniques, but they may feel threatened if they think that process 
data will be used by others to make judgments about their teaching ability. 
Whenever process data are used in summative reports, it is a good idea to omit 
the names of individual teachers and to emphasize how the process data were 
used to improve instruction. Su.ch an approach will remove the emphasis from 
evaluating teachers, but at the same time make decision makers aware that the 
program people (administrators, teachers, internal evaluator) are taking system- 
^ atic steps to evaluate themselves and to improve the program. 
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BASIC REFERENCE on Observation Instruments 



The following book provides descriptive information about 99 
systenns for observing, recording, and analyzing teacher and 
learner behavior. The systems are des^cribed in terms of: the 
number and type of subjects observed, the data collection 
method, the type of behavior upon which the instrument focuses, 
the type of coding units used, the personnel needed, and the 
reported uses of the system. This book is a must for the reference 
library of the process evaluator. 

Simon, Anita and E. Gil Boyer. Mirrors for Behavior III. An 
Anthology of Observation Instruments. Wyncote, Pa.: 
Communication Materials Center, 1974. 



The use of process data for program evaluation purposes can best be 
discussed by focusing on a couple of specific approaches to systematic observation 
techniques and classroom climate analysis. It goes without saying, of course, 
that any system or instrument will reflect the theoretical orientation and 
educational values of the developer; and therefore, the field evaluator will want 
to make certain that instrument(s) which he selects are in agreement with the 
philosophy and objectives of the program being evaluated. 

The most highly developed and widely used systems for gathering observational 
data is the Flanders Interaction Analysis System (See Basic Reference below). 
This system focuses on the distinction between direct and indirect teacher 
influence in the classroom, the underlying assumption being that the first step 
in modifying one's teaching behaviors is to more fully understand how the 
teacher is influencing the learning situation. Implicit in the target behaviors of 
this system (i.e., direct and indirect influence) is Flanders' belief that the 
teacher should develop the capability to make his or her own behaviors appropriate 
to the requirements of particular learning situations. 

Flanders' categories for an-cilyzing verbal interaction are shown in Figure 8. 
A trained observer (or teacher listening to a tape recording of her own classroom)- 
writes down a category number every three seconds throughout the course of 
a given activity. These numbers are then recorded on a data matrix and various 
statistical formulas are followed that lead to the discovery of certain patterns 
of teacher behavior. Although Flanders points out that only the individual 
teacher can make final decisions about what behavior is good or bad, desirable 
or undesirable, most of the research which has been carried out on this system 
seems to suggest that indirect teacher influence leads to the type of learning 
conditions which are frequently emphasized in programs for the gifted and 
talented. Indirect teaching generally has been found to result in higher student 
spontaneity, initiative, and contributions to problem solving situations. 
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Figures 

Summary of Categories for the Flanders Interaction Analysis System* 
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ACCEPTS FEELING: accepts and clarifies the feeling 
tone of the students in a nonthreatening manner. Feelings 
may be positive or negative. Predicting or recalling feelings 
is included. 

PRAISES OR ENCOURAGES: praises or encourages 
student action or behavior. Jokes that release tension, but 
not at the expense of another individual; nodding head, or 
saying "um hm?" or "go on" are included. 

ACCEPTS OR USES IDEAS OF STUDENTS: clarifying, 
building, or developing ideas suggested by a student. As 
teacher brings more of his own ideas into play, shift to 
Catego!^y 5. 

ASKS QUESTIONS: asking a question about content or 
procedure with the intent that a student answer. 



5. * LECTURING: giving facts or opinions about content or 

procedures; expressing his own ideas, asking rhetorical 
questions. 

6. * GIVING DIRECTIONS: Directions, commands, or orders 

with which a student is expected to comply. 

7. * CRITICIZING OR JUSTIFYING A UTHORITY: statements 

intended to change student behavior from nonacceptable to 
acceptable pattern; bawling someone out; stating why the 
teacher is doing what he is doing; extreme self-reference. 
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8. * STUDENT TALK^RESPONSE: talk by students in 

response to teacher. Teacher initiates the contact or solicits 
student statement. 

9. * STUDENT TALK-INITIATION: talk by students, which 

they initiate. If "calling on" student is only to indicate who 
may talk next, observer must decide whether student wanted 
to talk. If he did, use this category. 



1 0.* SILENCE OR CONFUSION: pauses, short periods of 
silence, and periods of confusion in which communication 
cannot be understood by the observer. 



There is NO scale implied by these numbers. Each number is classificatory; it designates 
a particular kind of communication event. To write these numbers down during observation 
is to enumerate-not to judge a position on a scale. 



* Amidonand Flanders, 1971, p. 14 
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Higher achievement, less dependence, and greater self-direction were usually 
found in the classroonns of teachers who were classified as highly indirect. Thus, 
we see in.the Flanders Systenn a technique that has great relevance to the task 
of evaluating the processes which are frequently encouraged in gifted education 
The systenn is especially useful in gathering data at lower grade levels because 
stujcJents are not required to connplete questionnaires or rating scales. 



BASIC REFERENCE on the Flanders System 



A great deal has been written on the Flanders. Systenn, but the 
following reference is the best "how-to-do-it" guide. Careful study 
of this little nnanual and a snnall annount of practice will provide 
the training you need to gather and analyze interaction data. 

Annidon, E.J. and N. A. Flanders. . The Role of the Teacher in 
the Classroom: A Manual for Understanding and Improving 
Teacher Classroom Behavior, Minneapolis: Association for 
Productive Teaching (5408 Chicago Avenue South, 
Minneapolis, Minn. 55417), 1971. 



The problenns associated with process evaluation and especially with 
observational techniques are well-known by nndst educators. Observer bias, 
lack of inter-judge reliability, a linnited sampling of teacher behavior, and the 
annount of tinne and nnoney involved have frequently been cited as reasons for 
approaching process studies with a great deal of caution. An "observer in the 
classroonn nnay result in a highly artificial situation. If the observer drops in 
unannounced, the teacher nnight feel threatened and react in an atypical 
nnanner. On the other hand, if the teacher knows exactly when the evaluator 
is coming (a condition that has been written into some union contracts), the 
evaluator may very well end up observing a "showcase" lesson. The best way 
to avoid these pitfalls-is to inform the staff that observation will be taking 
place over a given segment of time and they can expect someone to be 
visiting their classroom periodically. This approach lessens the shock of 
unexpected visitation and increases the chances of observing a classroom 
under natural conditions. Of course, the best approach is to make teachers 
sensitive to the role of formative evaluatioii, and thereby gain their cooperation 
and perhaps even enthusiasm for obtaining feedback that will help to 
improve the learning conditions in their classroom. 

A second approach to the process evaluation offers some intriguing 
possibilities for overcoming some of the problems discussed above. Rather 
than having one or two observers in the classroom for a very limited period 
of time, the questionnaire or rating scale approach is based on observations 
made by numerous observers who are in the classroom all of the time. These 
observers are, of course, the students themselves. 
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The Class Activities Questionnaire (CAQ) was developed by Joe M. Steele 
as part of the instrumentation that was used to evaluate the Illinois Gifted 
Program. Both cognitive and affective dimensions of the instructional climate 
are measured through a thirty-item instrument that is completed by both students 
and teachers. Steele (1969) describes' the dimensions of the instrument as follows: 
1 & 2) Lower Thought Processes and Higher Thought Processes 
assess the dimensions of cognitive emphasis. This part of 
the CAQ is based on Bloom's Taxonomy. Each higher level 
of thinking is believed to involve the use of all the lower 
levels. The difference between lower and higher levels is one 
of complexity. There can be a range of difficulty of activities 
at each level of thinking. 

3) The Classroom Focus dimension assesses whether focus is 
on the teacher as information-giver with students having a 
passive role, or on the students being given an active role 
with the teacher being the facilitator. 

4) The Classroom Climate dimension deals with the effective 
domain. It assesses factors such as how relaxed and open 
the class is. 

5) The Student Opinions dimension represents mini-interviews 
with each student on the best things and changes to make 
in the class. 

An overview of the CAQ is presented in Figure 9. One of the innovative 
features of this instrument is the way in which it is scored. Student responses 
are considered to be the Actual emphasis that is placed on each of the factors 
and these are summarized on the printout (See Figure 10) with the letter "A." 
Teachers fill out two copies of the questionnaire. On one copy they record the 
Intended or Ideal amount of emphasis that they would like to place on each 
factor (signified by the letter "I"). The second copy records their Predicted 
(P) emphasis. In analyzing their results teachers can determine how successful 
they were in achieving their ideal behavior and how accurately they estimated 
the way students viewed the class. Discrepancies can be used as the basis for 
improving instruction through self-analysis and/or inservice training activities. 

The fact that the CAQ is based, in part, on the Taxonomies of Educational 
Objectives makes it particularly relevant to process evaluation in gifted education. 
As was indicated in the early pages of this Guidebook, I firmly believe that a 
system such as the Taxonomies helps us to focus our attention on those higher 
cognitive and affective processes which should be given major emphasis in programs 
for the gifted and talented. 
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Figure 9 

STRUCTURE OF THE CLASS ACTIVITIES QUESTIONNAIRE (CAO)* 

The CAQ assesses five major Dimensions of instructional climate, as noted in the left- . 
hand column. Each of these dimensions is composed of a number of Factors which in turn 
are usually represented by several items in the questionnaire. (The Cognitive Dimensions 
are based on Bloom's Taxonomy.) 



DIMENSIONS 




FACTORS 


BRIEF DESCRIPTIONS (Items not shown) 




1. 


Memory: 


Activities calling for recall or recognition of infor- 


LOWER 






mation presented. 


THOUGHT 


2. 


Translation: 


Activities calling for paraphrasing or expressing 


PROCESSES 






information in a different symbolic form. 




3. 


Interpretation: 


Activities calling for recognition of relationships and 
seeing implications of information. 




4. 


Application: 


Activities calling for selection of appropriate methods 
and performance of operations required by problem 
situations. 


HIGHER 


5. 


Analysis: 


Activities calling for recognition of the structure of 


THOUGHT 






material, including the conaitions that affect the way 


PROCESSES 






it fits together. 




6. 


Synthesis: 


Activities calling for the generation of new ideas and 
solutions. 




7. 


Evaluation: 


Activities calling for development and application of 
a set of standards for judging worth. 




8. 


' Discussions: 


Student opportunity for and involvement in class 


CLASSROOM 
FOCUS 


Q 


Toct /fn raHo 
1 CO 1/ O [ aUc: 

Stress: 


discussion. 

Hinh nrp'5'5iirp tn nrndurp tp?irhpr-'?p|prtpd rin'^wpr'? 

for a grade. 




10. 


Lecture: 


Teacher role is information-giver with a passive, 
listening role for students. 




11. 


Enthusiasm: " 


Student excitement and involvement in class activities. 




12. 


Independence: 


*r-. 

Tolerance for an encouragement of- student initiative. 




13. 


Divergence: 


Tolerance for and encouragement of many solutions 
to problems. 


CLASSROOM 
CLIMATE 


14. 
15. 


Humor: 
Ideas Valued: 


Allowance for joking and laughter in the classroom. 
Ideas are seen as more important than grades. 




16. 


Ideas Enjoyed: 


Subject matter is seen as interesting and enjoyable. 




17. 


Teacher Talk: 


Proportion of class time consumed by teacher talk. 




18. 


Homework: 


Weekly amount of outside preparation for class. 


STUDENT 


19. 


Qualities: 


Students' view of the best things about the class. 


OPINIONS 


20. 


"Deficiencies: 


Students' view of things that need changing about 
the class. 



* Steele, 1969 
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Figure 10 

Sample Computer Printout of Class 
Activities Questionnaire Summary 



CITY: SEYMOUR JR. HIGH 



TLACHEK: 



SUOJCCT: SCUNCi: 



PERIOD: 0 GRADC LEVEL: S 



CAQ: TEACHER'S INTENDED EMPHASIS, PREDICTED RESPONSE, AND SUMMARY OF STUDENT RESPONSES PACE 4 OF 5 



LEVELS OF THINKING 



TEACHER'S INTEMOED (I) AND PREDICTED (P) EMPHASIS 
COMPARED TO ACTUAL (A) EMPHASIS SEEN BY STUDENTS 
INCONSISTENT NONE SOME MUCH 



LOWER 
LEVEL 
THOUGHT 
PROCESSES 



MEMORY 

TRANSLATION 

INTERPRETATION 



A 

A I P 



I P 



A P 



APPLICATION 






A 


P 




I 




HIGHER ^ 

LtVEL AMAivctC 

THOUGHT «w«LTbib 
PROCESSES 


. _.5, _ , 




A I 


P 







SYNTHESIS 


-. 6 








A 


IP. 




EVALUATION 


7 A 






P 




I 




CLASSROOM CONDITIONS 
















DISCUSSION OPPORTUNITY 


8 




A I 


P 








TEST/GRADE STRESS 


9 


A I P 












LECTURE 


10 


A I P 












ENTHUSIASM 


n 








A 


I P 




INDEPENDENCE 


12 








A 


I P 




DIVERGENCE 


13 








A 


I f» 




HUMOR 


14 








A 


I P 




IDEAS VALUED OVER GRADES 


IS 




A 






I P 




ENUOYMENT OF IDEAS 


16 








A 


I P 





PERCENTAGE OF TEACHER TALK 17 
AVE^rATJEi^PREPARATION TIME PER WEEK 



STUOENT-ESTIHATE TEACHER-ESTIMATE TEACHER-IDEAL 
25 25 25 
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AVERAGE .PR£PAP.ATION 
TIME PER WEEK 



18 



0 HR 



5.' 



0.5HR 



1.5HR 



BASIC REFERENCE on the Class Activities Questionnaire 



I have administered the CAQ in a number of evaluations of gifted 
programs and have been thoroughly impressed with its usefulness. 
The teachers eagerly looked forward to receiving their results and 
used them as a guide in the selection of consultants for inservice 
training. The cost is reasonable (about $10.00 per classroom) and 
includes computer scoring and printouts. 

Steele, Joe M. Dimensions of the Class Activities Questionnaire, 

Urbana, Illinois: Center for Instructional Research and 
Curriculum Evaluation, University of Illinois, 1969. 

Information about the instrument and scoring service can be 
obtained by writing to: 

Dr. Robert Rosemier 
520 College View Court 
Northern Illinois University 
De Kalb, Illinois 601 1 5 



BASIC REFERENCE on other Classroom Climate Questionnaires 



Descriptions of six additional classroom climate questionnaires can 
be found in the following reference. This article also includes an 
excellent summary of research on relating classroom climate to 
student outcomes. 

Nielsen, H. Dean, and Diana H. Kirk. "Classroom Climates/' 
In H. J. Walberg (Ed.), Evaluating Educational Performance, 
Berkeley, Calif.: McCutchan Publishing Co., 1974. 



E. 'Presage or Intrinsic Evaluation 

Some evaluators would argue that due to deficiencies in instruments that 
measure products (payoff), especially higher level cognitive and affective products, 
it is necessary to look for other sources of evaluative data. Presage or intrinsic 
evaluation focuses on factors which are assumed to have a significant inn,pact on 
outcomes or products. Thus, intrinsic factors may be thought of as the purposefully 
planned activities that are designed to bring about changes in student perforr},ance. 
According to Scriven (1967), persons who advocate this approach in curriculum 
evaluation are likely to be concerned with certain "qualities of a curriculum such 
as elegance, modernity, structure, integrity, readiness considerations, etc., which 
can best be judged by looking at the materials directly" (p. 54 ). Renzulli and Ward 
(1969) have used the presage approach more broadly in the development of DESDEG 
by pointing out several dimensions of a program that can be studied through the 
assessment of information that has an assumed relationship to program quality. 
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The major problem with presage or intrinsic evaluation is the logical jump 
that must be made from intrinsic factors to program outcomes. Although out- 
standing teachers, low teacher-pupil ratios, and carefully developed learning 
activities would seem to result in generally better products, the evaluator is frequently 

with the haunting feeling that some product and process data are necessary in 
order to indicate that pay-off is really taking place. Indeed, very few contemporary 
evaluation theorists would advocate a wholly presage approach, and yet, this 
concept is of value in considering the assessment of programs for the gifted and 
talented because the outcome objectives of such programs are oftentimes not 
easily measured by existing instruments. The presage approach also offers a 
useful model of evaluating non-product dimensions of a program. For example, 
in the DESDEG model Renzulli and Ward have developed several forms that 
help to provide an analytic look at identification systems. The comprehensiveness 
of screening and placement procedures, the variety of criteria used in identification, 
and the proportion of students selected at each grade level are revealed through 
the use of these forms. The forms force a breakdown of the information and 
thus enable the evaluator to see the identification system more clearly and to 
ask more meaningful questions. Hopefully, a clearer pictureof all aspects of the 
identification process will help the evaluator come to more accurate judgments. 

The presage or intrinsic approach is probably more in keeping with Stake's 
(1967) belief that careful and accurate description is a necessary prerequisite to 
the judgmental process. As has been indicated earlier, information (data) does^ 
not in and of itself render opinions; and in the final analysis, it is people who must 
make judgments. The presage approach can facilitate the judgment process by 
helping to present information in its clearest and most useful format. In the 
chapter that follows a combination of models and approaches will be advocated; 
however, we vyill draw heavily on the presage concept as it relates to presenting 
descriptive information. 

F. Other Models and Concepts 

Before ending this section we should again reemphazise that no single 
model or approach is sufficient for solving all of the problems that are likely 
to occur in a wide and frequently complicated variety of evaluation situations. 
Each approach has certain strengths and limitations and it is up to the field 
evaluator to size up his or her particular problem and to apply a combination 
of the most appropriate methods. 

We also should not leave this section without pointing out that additional 
models and methods can be found in ever-expanding literature on evaluation. 
A recent volume in the American Educational Research Association Monograph 
Series on Curriculum Evaluation (1974) presents four reports on educational 
evaluation that deal with Anthropological, Economic, Narrative, and Portral 
approaches. Worthen and Sanders (1973) and Borich (1974) have included 
several models in their very valuable reference books on evaluation theory, and 
Michael Scriven (several years ahead of his time, as usual) has recently developed 
an entirely new concept called goal-free evaluation. But the models and concepts 
which we have included in this section represent "the basics/' and as such, they 
should provide the necessary background for additional reading on evaluation 
theory and practice. 
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CHAPTER THREE: Designing an Evaluation 
for Your Program 

This chapter represents a synthesis and application of nnuch of the infornnation 
that has been discussed in the first two chapters. An attennpt has been nnade to 
provide specific suggestions for planning and carrying out an evaluation study, but 
it is innportant to keep in mind that no single nnodel or fornnula can be prescribed 
for the nnany different progranns and circumstances under which evaluative studies 
are conducted. The theories, models, and concepts have, of course, greatly influenced 
the direction that this chapter will take, but the realities encountered through 
actual experience in evaluating programs for the gifted have provided the major 
guidance for the suggestions. 

Part I will deal with a political reality that should be considered at the start 
of any evaluation study. The most sophisticated evaluation design and the most 
careful procedures will not serve the purposes of evaluation if key decision makers 
do not get the information that they require. For this reason we will begin by 
analyzing the decision-making process and offering some suggestions for clarifying 
the evaluator's relationship with decision makers. 

Part 1 1 is concerned with specific procedures that can be followed in planning 
and carrying out an evaluation. Although this section might be described as an 
• evaluation model, it may be thought of as a management system into which the 
ideas of many of the model builders can be incorporated. In developing this section, 
an effort has been made to avoid dragging one through the basic content of a 
tests and measurements course; but it is difficult to deal with theicentral topic of 
instrumentation without getting into a little bit of the measurement jargon. 
Throughout this section we have attempted to point out the technical competencies 
in measurement and statistics about which the evaluator should be generally 
knowledgeable about; but at the same time, we have frequently suggested that 
the services of specialists in these areas be sought. 

I. Examining the Decision-Making Process 

Why is the evaluation being conducted? 

Who will receive the evaluation information and what will they do 
with it? 

What kind of information do key decision makers want? 

Who should carry out the evaluation? 

Before we can begin to plan a truly useful evaluation study it is important 
to consider the above questions. They can be used as guides in analyzing two 
important dimensions of the decision making process. The first dimension is 
concerned with certain "agreements" that should be reached between the 
evaluator and decision-making individuals or groups. The second dimension 
deals with identifying areas where decisions might be made as a result of an 
evaluation study. We will use two hypothetical examples to help illustrate these 
two points. 
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I V 



A. A Board of Education Wants fo Make a Go/No-Go Decision 

Suppose that a Board of Education wants to evaluate its gifted program 
in order to deternnine whether or not it should be continued. The evaluation 
will result in a sinnple decision— continue (Go) or discontinue (No-Go) the 
progrann. Let us further suppose that the board is "test-score oriented" and, by 
and large, they are nnainly interested in pretest/post-test comparisions of 
student achievennent. Finally, let us assunne that when test results are presented 
the board is likely to ask the "So what?" question. That is, the changes in test 
scores for gifted students will not, in and of thennselves, satisfy the board unless 
they have sonne basis for nnaking connparisons. This situation quite obviously 
requires a research design or highly controlled sunnnnative evaluation study 
with nnajor ennphasis on the connparative analysis of product or outcome data. 

Since one of the most difficult tasks in this evaluation will be selecting 
appropriate measuring instruments and analyzing the result, it would be wise 
to choose an evaluator with a strong background in tests and measurements 
and research methodology. In view of the board's requirements, the evaluator 
must be knowledgeable about these topics .so that he can present his require- 
ments for carrying out a study that will be satisfactory to them. He may, for 
example, decide that he must randomly assign students to the gifted program 
and control groups; or that he must administer a great number of tests to both 
students in the program and the general school population. The important 
issue here is that decision makers should be aware of the consequences of 
their requirements so far as the needs of the evaluator are concerned. And this 
awareness should cause them to grant necessary permissions and make available 
appropriate resources. 
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A. Step I: Front-End Analysis 

The purpose of Front-End Analysis is to help the evaluator identify "Key 
Features" in programs for the gifted and talented. Key Features may be thought 
of as major factors or variables that contribute to the effectiveness of a program. 
Before an evaluator can begin to select instruments and gather data it is important 
to determine which factors influence the program's operation and contribute 
most to an understanding of it. It is also important to learn what types of infor- 
mation are of major concern to various "Prime Interest Groups."tPrime Interest 
Groups consist of people who have a direct or indirect interest in the program 
being evaluated. Quite obviously, these groups will amost always include students, 
parents, teachers, administrators, and school board members. But, depending on 
the nature of the program. Prime Interest Groups may also include persons who 
are involved in the program indirectly. If, for example, a program draws heavily 
on the services of community businesses, cultural facilities, or agencies, the persons 
responsible for arranging and/or providing these services should be considered 
Prime Interest Groups. It is quite likely that many of the Prime Interest Groups 
will share some of the same concerns, but each group will also have some relatively 
unique interests. Thus, for example, several groups may be interested ia student 
achievement but school board members may want information on cost effective- 
ness. Administrators may be interested in learning how the gifted program affects 
the attitudes of non-program teachers and students; and teachers may want to 
know if certain curricular materials or teaching techniques are favorably received 
by students. One of the early jobs of the evaluator is to ferret out the major 
concerns of each of these groups. 

Careful work in the Front-End Analysis step will help the evaluator focus 
his efforts on the most relevant concerns of each group. The information 
obtained at this step gives direction to the difficult task of instrument selection 
and development and also serves as a guide in developing a time-line and preparing 
evaluation reports. Please keep in mind that Front-End Analysis is essentially an 
"input" or information gathering activity, and therefore the evaluator should 
avoid making judgments at this stage of his work. The main objective is to become 
thoroughly familiar with the program so that meaningful Key Features can be 
identified. 

The four procedures used in Front-End Analysis are described below. Although 
the procedures can be carried out simultaneously, experience has shown that more 
meaningful information can be obtained from the Interview and Visitation techniques 
if they follow the first two procedures. 

1 , Review of Documents. The first step in Front-End Analysis is a careful 
and comprehensive review of all written material relating to the program. These 
documents might include: 

a. Statements of Philosophy and Goals 

b. Guidelines and Instruments Used in Student Identification 

c. Curriculum Guides and Materials 

d. Descriptions of the Student Population 

e. Lists of Behavioral Objectives and Learning Activities ' 

f. Descriptions of the Staff and Criteria for Their Selection 



g. Project Proposals, Budgets, Time and Effort Reports 

h. Descriptions of Facilities, Equipnnent, and Support Services 

i. State Department of Education Reg^jlations Governing the Program 
j. Previous Evaluation Reports 

k. Descriptions of Inservice Training Activities 

1. ' Newsletters, Newspaper Releases, and Public Relations Brochures 

m. Samples of Students' Work 

n. Methods for Reporting Student Progress 

o. Diagrams or Charts that depict the Administrative Hierarchy or 
Chain of Command 

p. Superintendent's Annual Reports 

As the evaluator reviews program documents he should make careful notes 
about areas where he would like to raise further questions, obtain clarifications, 
or note discrepancies between the content of two or more pieces of information. 
Older and more highly developed programs will no doubt have a larger number and 
variety of documents, and thus, it may be necessary to review them selectively. 
When documents are relatively limited, as is frequently the case with new programs 
the evaluator will have to depend more heavily on the methods suggested below for 
input information. 

2. Open-Ended Input Questionnaires. A representative sample of each 
Prime Interest Group should be asked to complete an open-ended questionnaire 
such as the one presented in Figure 1 3. The major feature of an input questionnaire 
is its lack of structure and the freedom it allows persons to comment about anything 
they would like. A good way to discover the important values of respondents is 
simply ask what they would look at if they were evaluating the program. Whenever 
possible, questionnaires should be kept anonoymous and the directions should 
make it clear that no attempt will be made to identify persons completing questionnaires. 

p Input questionnaires should be analyzed separately for each Prime Interest 
Group so that you can compare and contrast the concerns of the respective groups. 
A good way to analyze this type of questionnaire is to develop a color-code system 
and simply underline similar key words or phrases with a certain color pen. For 
example, if statement dealing with "snobbishness" or "elitism" occur I might 
underline them in red. After all the questionnaires have been analyzed I can spread them 
out quickly and get a feel for the most frequently cited areas of concern. Sampling 
is a doubly important concern when distributing and analyzing questionnaires. 
First, you must have a fair porportion (40 to 60%) of returns from each group 
to insure a representative sample of the thinking of that group. Second, comments 
relating to particular areas of concern should occur with enough frequency to 
suggest that it is not a solitary attitude or grievance. (The extent and depth of 
concerns can be probed in the next procedure.) 
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Figure 13 
Sample Input Questionnaire 



Hamden-New Haven Cooperative Education Center 

INDEPENDENT STUDY PROGRAM 

The coordinators of the Independent Study Progrann (ISP) 
have asked nne to conduct an evaluation that will help thenn to 
determine the effectiveness of the progrann. The first step in the 
evaluational nnodel that I will be using is to identify the nnajor 
areas of concern of students, faculty nnennbers, and other persons 
connected with the progrann. These areas of concern nnight relate 
to course content, scheduling, facilities, attitudes of regular 
students toward ISP students, etc. 

I would like to obtain your assistance in this initial process 
by asking you to list the major questions that you would like to 
have answered in an evaluation of the ISP, Please list your 
questions or areas of concern in the space below and use the back 
of this page if you need nnore space. Please do not sigri your name 
to this questionnaire. No attennpt will be nnade to identify the 
person connpleting this fornn. 

Thank you for your cooperation, 

Joseph S. Renzulli 
University of Connecticut 
Box U-64 

Storrs, Conn. 06268 



Please check the space below that best describes your relationship 
with the progrann: 

ISP Student Guidance Counselor Departnnent 

" ~ Head 

ISP Faculty Adnninistrator Advisory 

Board' Mennber 

Parent 

, Other (Specify) 



3. Interviews With Representatives of Prime Interest Groups. Interviews 
should begin with the director and persons who were involved in the planning 
phase of the progrann. These persons will nnore than likely help you to gain an 
understanding of the way in which the progrann was ideally conceived, whereas 
interviews with teachers and students will probably deal nnore closely with the 
way in which the progrann is actually operating. The evaluator's first face-to- 
face contact with people is extrennely innportant, and it is at this point that he 
nnust gain their trust and cooperation. As was indicated in Chapter One, the 
evaluator can quickly be perceived as the "enenny" or the representative of 
"the oppressor" unless favorable rapport is developed. Rapport and trust can 
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be developed more easily when the evaluator assumes the role of an ombudsman, 
or person who represents "the people" in making their interests and concerns 
known to administrators and policy makers. In Front-End Analysis interviews, 
especially in a formative approach, the evaluator should essentially be asking: 
"How can I help you?" "What information will help to make the program 
better for you?" "What are the things that are bothering you?" It is important, 
of co^jrse, to assure the persons being interviewed that complete anonymity 
will be maintained. An informal and relaxed setting isabsoutely necessary, 
and small groups.or rap sessions are less threatening to students than being 
called in for a formal interview. It almost takes a sixth sense to determine if a 
tape recorder or too much note-taking are posing a threat to persons being 
interviewed. It is always a good idea to explain beforehand that tapes or notes 
are solely for the purpose of helping you remember what has been said and that 
no one but you will have access to the tapes or notes. 

In conducting interviews the evaluator should use information obtained 
from reviewing documents and open-ended questionnaires as cues or take 
off points to deeper levels of understanding. "Are the objectives realistic?" 
"Is flexibility and innovation really encouraged?" "Is there too much bureau- 
cratic red tape?" "Are students rea/// free to select whatever topics they 
would like?" The interviews should also help the evaluator determine if 
statements on the open-ended questionnaires are relatively limited or wide- 
spread concerns. "Do you think that a lot of people feel that the program 
fosters elitism?" "Do you think that a lot of kids in the program are really 
goofing off?" "Does the principal always give teachers a hard time when they 
ask permission for students to leave the building?" When strong feelings are 
expressed with regard to a particular issue you might ask, "Do you think that 
this would be a good question to ask all (students, teachers, parents, etc.) on 
a written questionnaire?" Asking the right questions in input interviews is 
difficult because you are not evaluating but rather attempting to identify areas 
that will help to structure subsequent evaluative activities. Do not be concerned 
if you are not getting a great many facts and figures— these will come later 
when you develop more structured data-gathering instruments. Remember, 
Front-End Analysis is for the evaluator's orientation rather than for making 
judgments. 



"would you CMAWWlTtClTE Yt)Df? TOlHON flS ' 
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4. Observation. A fourth procedure in Front-End Analysis is simply 
observing the program in action and trying to "get inside" it by viewing it 
from the perspective of students and teachers. Understanding the day-to-day 
operation of the program will help to clarify and verify som^ of the concerns 
identified through previous input procedures. This type of "reality orientation" 
will give you a much better perspective of the-way-it-is rather than the ways 
in which it has been described and discussed. 

B. Step II: Synthesis of Input Information and Instrument Development 

At the conclusion of Front-End Analysis the evaluator should be able to 
list the major concerns of each Prime Interest Group. These concerns should 
be classified and organized according to similarities among the groups and 
the list which evolves should make up the Key Features upon which the 
evaluation will focus. (Note: We will come back to the Prime Interest Groups 
in a later section.) As has been indicated earlier, differences among programs 
and the various concerns of decision makers will undoubtedly result in 
different Key Features from program to program; however, certain areas will 
probably be identified in most evaluation situations. Student performance, for 
example, is an almost universal concern of Prime Interest Groups, but other 
Key Features have been identified in research studies (See the following BASIC 
REFERENCE) and therefore each program should be analyzed individually 
to determine Key Features of local importance. 



BASIC REFERENCE on Identifying Key Features 



A systematic procedure for identifying Key Features in programs 
for the gifted was the subject of the following study. It was this 
study that provided the areas around which DESDEG is organized 
(See. Figure 7). 

Renzulli, Joseph S. "Identifying Key Features in Programs 
for the Gifted." Exceptional Children, 1968, 35, 21 7-221 . 
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Figure 14 

Matrix of Key Features and Sources of Data 

Key Features 



Student 



Levels of Thinking 
and 



Attitudes Toward Identification 



Students 



Program 
Teachers 



Parents 



a 

o 

I 

Co 



Student 

Selection 

Committee 

(Including 

Records) 



Non- 
Program 
Teachers 



Consultants 



Building 

Principals 

and 

Coordinators 



Secretaries 



Growth Classroom Conditions Program 


Procedures 


Etc. 


Pre- and Post-Tests 
of Creativity, 
Critical Think- 
ing, etc. 


Class Activities 
Questionnaire 

Interviews 


Questionnaires 
Interviews (Random 
Sample) 






Evaluation of 

Student Growth 
Forms (A Struc- 
tured Anecdotal 
Report)* 


Class Activities 
Questionnaire 

Logs 


Interviews 


Time and Effort 

Reports 
Follow-up 

Questionnaire 








Questionnaires 
Interviews (Random 
Sample) 


Follow-up 

Questionnaire 










\ iiTic anu C 1 1 Ul L 

Reports 

Rating Scale (on 
Usefulness of 
Information) 

Interviews 

Analysis of Records 




Rating Scale 




Questionnaires 
Interviews (Random 
Sample) 


Time and Effort 

Reports 
Follow-up 

Question naire 




Student Product 
Rating Form 












Questionnaires 


Interviews 
"Problems" Log 


Time and Effort 

Reports 
"Problems" Log 










Time and Effort 
Reports 





*See Fit'ure 17 
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After the Key Features have been identified they should be listed along 
one dimension of a chart such as the one presented in Figure 14. The 
evaluator must now ask himself two questions: 

1 . What types of instruments and/or techniques will 
provide information relevant to each Key Feature? 

2. From whom can this information be obtained? 

The answers to these questions provide the information necessary for 
completing the other dimension of the chart (Sources of Data) and filling 
in the actual content which consists of the instruments that will be used to 
gather data related to each Key Feature. Following the completion of this 
matrix the evaluator should then prepare a Data Collection and Analysis 
Guide for each Key Feature. An example of such a guide is presented in 
Figure 15. This particular example is from an evaluation study that attempted 
to compare various approaches to the identification process. As can be seen 
in Figure 1 5, the first column provides a breakdown of specific activities 
that are involved in Key Feature i: Student Identification Procedures. 
Subsequent columns list the sources of data, how the data will be gathered, 
the data analysis procedure for each instrument, and the dates when informati 
will be obtained. Thus, the Key Features matrix (Figure 14) provides us with 
an overview of the entire evaluation while the Data Collection and Analysis 
Guides (Figure 15) provide more detailed direction for gathering and 
processing information relevant to each Key Feature. As specific instruments 
are identified, general terms such as "Questionnaire" and "Test" can be 
replaced by the names of actual instruments. 
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1 . Instrument Selection. We have now reached what is perhaps the 
most difficult task in evaluating programs for the gifted and talented — 
selecting and constructing appropriate data gathering instruments. Although 
it is beyond the scope of this Guidebook to cover the essentials of a course 
in measurement or instrument development, an attempt will be made to 
provide some basic guidelines and references for instrument selection and 
construction. It is at this point that the knowledge and competence of the 
evaluator will weigh heavily in the quality of the evaluation. Therefore, 
it is recommended that if the field evaluator does not have a strong back- 
ground in tests and measurements, he should obtain the assistance of a 
consultant with extensive experience in this area of specialization. The testing 
business has experienced tremendous growth in size and complexity over 
the past several years and it is unrealistic to expect a non-specialist to be 
aware of the wide variety of available instruments and the technical problems 
that are involved in testing, if the evaluator is unable to secure the assistance 
of a consultant, it is recommended that he familiarize himself with the 
principles of testing by reviewing one or more of the books in this area that 
have been included in the bibliography on Measurement, Statistics, and 
Research Design. 

To begin with, a basic rule to follow is that ths evaluator should always 
explore the availability of existing instruments before he attempts to develop 
his-own materials. This is especially true for tests that measure student 
performance. The construction of valid and reliable tests is an extremely 
time consuming and expensive affair that is best left to the professionals. 
Even questionnaires and rating scales should possess respectable levels of 
reliability and validity, and therefore it is suggested that a thorough review 
of existing instruments be conducted before instrument construction is 
considered. A good place to begin this review is in the annotated bibliography 
of Instrument Sourcebooks that is'included at the end of xh\s Guidebook: 
These sourcebooks describe thousands of instruments that have been developed 
by commercial publishers, researchers, and practitioners. The instruments 
are classified according to the areas which they measure and it is generally 
quite easy to locate which instruments are available for assessing particular 
areas. Some of the sourcebooks include instruments in their entirety and 
many provide information about reliability, validity, age and grade level, 
cost, and administration time. 

The decision regarding which instrument to select should be guided by 
three basic criteria. First and foremost, we must determine the validity of 
the instrument for our particular situation, i.e., how adequately the instrument 
measures the specific educational goals or activities that we are attempting 
to evaluate. Most standardized tests and inventories include information on 
validity in their manuals. This information should be checked, and in addition, 
we should carefully review the content (specific items) of the instrument to 
make certain that it fits our purpose. Reliability of information should also 
be checked, but a highly reliable instrument is of no value whatsoever if it 
does not measure what we want it to measure; thus, validity is an absolutely 
essential requirement. Related to validity is the issue of usefulness of the 
rft: -«'ts in the decision-making process. Each instrument should contribute 
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information that will help us to make decisions that are related to one or more 
of the Key Features. A good way to estimate usefulness is to think ahead to 
what types of action can be taken if the results turn out in a certain way. 
If clearly positive or negative results warrant identifiable action/then chances 
are the instrument meets the usefulness criterion. 

The second criterion that should guide us in instrument selection is the 
appropriateness of the instrument for the population that is being examined. 
Factors such as level of reading difficulty, clarity of instructions/length and 
attractiveness of format are important considerations in selecting instruments 
for adults as well as children. Persons can quickly become frustrated or weary 
if an instrument is confusing, boring, or does not contain material which is 
relevant. Whenever possible, it is a good idea to field test an instrument on 
a small sample population in order to estimate what type of reaction you 
will get to the instrument itself rather than what it measures. 
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The third criterion which we should consider in instrument selection 
is simpty practicality. An instrument that is extremely time consuming, 
expensive to score, or requires the services of a highly trained examiner 
or observer may very well be an uneconomical selection. If large numbers 
of students are being tested or numerous questionnaires are being admin- 
istered, it would be wise to choose instruments that can be scored by 
machine. Practical requirements should, of course, be weighed in relation 
to the information that is necessary for decision making and the human 
and financial resources that are available in an evaluation study. It goes 
without saying that the practicality criterion should never be substituted 
for the more important criteria of validity and appropriateness. 

Evaluators who work on curriculum development projects frequently 
become involved in test construction, however, the work of most field 
evaluators is restricted to constructing questionnaires, rating scales, and 
other types of non-test instruments. But field evaluators usually are 
responsible for test selection and therefore they should be familiar with 
the following Basic References. 



BASIC REFERENCES on Tests in Print 



these two books provide comprehensive categorical listings 
of almost all commercially available tests that are printed in 
the English language. They are absolutely essential items in 
the libraries of professionaTevaluators. 

Buros, 0. K. (Ed.) Tests in Print Highland Park, New Jersey: 
Gryphon Press, 1961. 

Buros, 0. K. (Ed.) Tests in Print II. 
(Same Publisher), 1974. . 



2. Instrument Construction, Let us now turn our attention to some of 
the instrument construction activities in which the field evaluator is typically 
engaged. At the bottom of the second column in Figure 12 we note a break- 
down of the types of instruments that evaluators are usually called upon to 
construct. After the Key Features have been identified and the Activities 
Being Evaluated have been listed in the Data Collection and Analysis Guides 
(Figure 15), the evaluator must decide which types of instruments will 
effectively deliver the kinds of information that he is seeking. It is at this 
point that the evaluator should consult with a measurement specialist or 
review a basic book on testing in order to determine the advantages and 
disadvantages of each type of instrument.* Generally, questionnaires, rating 



* I have found Thorndike and Hagen's book, Measurement and Evaluation in Psychology 
and Education (Third Edition) to be one of the most useful and clearly written texts 
in this area. It is cited in The Basie Reference on page 1 1. 
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scales, and checklists are most useful for assessing attitudes toward a progrann 
or patricular connponents of a progrann. interview schedules may also be used 
to probe attitudes at greater depths and to determine some of the causes that 
give rise to certain attitudes. Later in this section we will also show how sono- 
grams have been used to evaluate the attitudes of non-program students toward 
students who are taking part in a gifted program. 

Logs, anecdotal recording system, inventories, and checklists can be used 
to evaluate various aspects of student performance. Again, we will provide 
examples of such instruments in subsequent parts of this section. Finally, 
observational systems are most useful for evaluating processes such as classroom 
interaction and the dynamic qualities of the learning environment. A discussion 
dealing with observational. systems and instruments was included in the Process 
Evaluation section of Chapter Two. 

Before we go on to describe some of the specific steps in instrument 
construction, it is important to point out that there are two approaches to the 
development of homemade instruments. The first and most frequently used 
approach is simply for the evaluator to sit down and design an instrument 
that seems to make sense because the items logically correspond to the activity 
being evaluated. In this case, we claim that the instrument has "face" validity 
because it appears to be an appropriate measure. When resources for evaluation 
are limited, face validity is often the only type of claim that an evaluator can 
make about the quality of an instrument. Although evaluators with a great 
deal of accumulated experience can undoubtedly develop'instruments with 
very good face validity, the value of many evaluation studies is often question- 
able because-the relative strength of key instruments has not been established. 

The second approach to the development of homemade instruments 
involves several systematic procedures that are designed to establish content 
validity, construct validity and reliability. These procedures resemble the first 
two steps in the Key Features System and they will be discussed in relation to 
the development of instruments that can be used to measure attitudes toward 
gifted programs. The same procedures can also be used to construct other types 
of instruments, but attitude measurement will serve as our example since this is 
the type of instrument that the evaluator .is most frequently called upon to 
construct. 

a. Measures of Attitudes Toward Gifted Programs, We will use an instrument 
entitled the Student Attitude Toward Independent Study Questionnaire (SATIS-Q) 
to review the steps involved in establishing validity and reliability. This instrument 
and an analysis of the results obtained from actually using it in a specific program 
are discussed here in their entirety for three reasons. First, the discussion provides 
a case study in instrument construction that can serve as a model for instruments 
that you may have to develop. Second, it provides an example of how to write-up 
the results of an attitude evaluation. And third, in the event that. you may want 
to adopt the SATIS-Q for your own purposes, the case study can be used to 
help support your rationale for such an adoption. 
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Case Study in Instrument Development 



The first step in constructing the SATIS-Q consisted of 
developing a list of relevant concerns and issues related to this 
particular nnethod of learning. This was done by reviewing the 
literature in this area and interviewing students, teachers, ' "--^ 
counselors, and adnninistrators who were involved in independent 
study progranns. Representative sannples of these groups of ■ 
persons were also asked to respond to an open-ended question- 
naire (See Appendix A). These "input" procedures helped to 
establish the content validity of the scale. Fronn infornnation 
derived through these procedures, an initial pool of 65 rating scale 
itenns was generated. Each itenn described a particular characteristic, 
objective, or anticipated benefit of independent study, and students 
responded by nnarking a Likert type scale which delineated five 
levels of quality for each progrann characteristic (Excellent, Above 
Average, Acceptable, Below Average, Extrennely Poor). 

The pilot instrunnent was adnninistered to 1 09 students who 
were enrolled in an independent study progrann for at least one 
year. The construct validity was exannined by generating an 
intercorrelation nnatrix for the response data ennploying a principal 
connponent 'factor analysis followed by an oblique rotation. An 
eight connponent solution accounted for 75 percent of the variance 
associated with the itenn interrelationships; however, three factors 
were dropped fronn the instrunnent because of the snnall nunnber 
of itenns that achieved loadings of .35 or higher. Thus, the final 
version of the instrunnent contained five factors and 27 items. 
The nature of each factor will be discussed in the section that 
follows. Table 1 contains the primary pattern matrix for the 27 
items that were retained in the SATIS-Q. The following five 
factors emerged: Influence of Independent Study on Motivation 
and Career, Freedom to Pursue Personal Interests, Effect of 
Independent Study on Study Habits and Thinking Processes, 
Degree to which Independent Study Helps to Fulfill Personal 
Objectives, and Degree of Challenge and Opportunity for Self- 
Expression. Table 2 contains the intercorrelations of the derived 
factors. Table 3 contains the number of items and the alpha internal 
consistency reliabilities for each derived factor. Although Table 2 
shows that some of the factors v/ere sufficiently correlated so they 
might be combined to increase reliability (e.g., Factors I and III, 
II and IV), the reliabilities reported in Table 3 indicate that the 
five scale scores can be used separately in assessing attitudes 
toward independent study. Since Factors I and III (r = .48) and II 
and IV (r = .41 ) are related, persons using the scale may wish to 
combine the factors to produce a more reliable composite score 
based upon a larger number of items. 
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Table 1 






Primary Pattern Matrix 


: Principal Component Solution 




and Obliguimax Transformation for Items Retained on the SATIS Q* 


ITPMQ 
1 1 CIVIO 


1 


II 


III IV 

III IV 


V 


1 

1 




?7 






2 




91 






3 




87 






4 








38 


5 






64 




6 






88 




7 








59 


8 


59 








9 








70 


10 


72 








11 


90 








12 


86 








13 


85 








14 






74 




1 ^ 






96 




16 






83 




17 






56 




18 


52 








19 


56 








20 








35 


21 








59 


22 


61 








23 


65 








24 






65 




25 






48 




26 




51 






27 ■ 






53 




* All entries multiplied by 100 to elimated decimals. 






Table 2 






Intercorrelations of Derived Factors on the SATIS-Q 






1 


II 


III IV 




II. 


.25 








III. 


.48 


.28 ■ 






IV. 


.33 


.41 


.36 




V. 


.38 


.29 


.37 .14 
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Table 3 



Number of Items and Alpha Internal Consistency 
Reliabilities for the SATIS Q 



FACTORS NUMBER ALP^A 

OF ITEMS RELIABILITY 

I. Influence of Independent Study 

on Motivation and Career 9 .90 

II. Freedonn to Pursue Personnal 

Interests 4 75 

III. Effect of Independent Study on 
Study Habits and Thinking 

Processes 5 87 

IV. Degree to Which Independent 
Study Helps to Fulfill Personal 

Objectives 4 79 

V. Degree of Challenge and Opportunity 

for Self-Expression 5 .74 



Description of Program 

The revised version of the SATIS-Q was administered to 196 
high school students who participated in the Independent Study 
Program for Gifted Students (ISP) sponsored by the Hamden-New 
Haven (Connecticut) Cooperative Education Center. Students who 
are enrolled in one of the four high schools served by the ISP may 
pursue one or two topics in lieu of a regular course or courses in 
the usual school program. Each student is assigned to a teacher or 
resource person who specializes in a particular content area, arid 
together they develop a proposal for study which serves as a guide 
for investigating a particular independent study topic. Many non- 
traditional topics are explored and students are frequently involved 
in innovative research that is sometimes conducted in out-of-school 
settings. 

The means, standard deviations, and the alpha internal 
consistency reliability coefficients for each factor on the SATIS-Q 
are presented in Table 4. An analysis of student attitudes for each 
of the five factors measured by the SATIS-Q follows. 
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Table 4 

Means, Standard Deviations, and Alpha Internal Consistency 
Reliability Coefficients for ISP Students on the SATIS Q 



SCALE I SCALE II SCALE III SCALE IV SCALE V TOTAL 



Number of Items 


9 


4 


5 


4 


5 


27 


Maximum Possible 
Score 


45 


20 


25 


20 


25 


.135 


Mean Score 


31.70 


16.03 


18.35 


15.17 


18.42 


99.67 


Standard Deviation 


6.81 


3.12 


3.70 


2.82 


4.07 


16.62 


Alpha 

Reliability 


0.83 


0.73 


0.82 


0.67 


0.70 


0.87 



Factor I : Influence of Independent Study on Motivation and 
Career. The nine items which define this factor indicate that the 
program is strongly influential in helping students to decide on the 
area(s) in which they would like to major in college. Students scor- 
ing high on this factor feel that the program is influential in helping 
them to determine the types of careers they would like to pursue. A 
large majority of students indicated that the ISP increased their 
excitement about learning and helped them become better evaluators 
of their own work. 

Factor 1 1 : Freedom to Pursue Personal Interests. This factor 
generally reflects the students' perceived opportunities to select 
independent study topics which are of personal interest and to 
pursue these topics to the extent desired. Persons who score high 
on this factor generally feel that independent study allows them 
freedom in both content selection and the degree to which they 
wish to delve into a topic. Student attitudes in this area are extremely 
favorable. Almost all of the participants rated the program very high 
on this factor and approximately two-thirds of the students 
indicated that the objectives of the ISP were in agreement with 
their own personal objectives. 

Factor III. Effect of Independent Study on Study Habits 
and Thinking Processes. This factor deals with such processes as 
critical thinking, organizing and focusing one's thoughts, and 
developing more effective study habits. High scores on this factor 
reflect a positive influence of independent study on processes 
that contribute to ef fective learning. On the average, sixty to 
seventy percent of the students in the ISP displayed very favorable 
attitudes toward the program as far as its effect on these processes 
was concerned. Almost 70 percent of the respondents indicated 
that the program helped them to become better acquainted with 
their preferred styles of learning. 

Factor IV: Degree to Which Independent Study Helps to 
Fulfill Personal Objectives. The four items defining this factor deal 
with the way in which the independent study approach to learning 
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is organized and how well the program meets personal objectives. 
High scores on this factor indicate that students are satisfied with 
the way in which the program is meeting their objectives. Between 
one-half and three quarters of the students involved in the ISP 
responded positively to items which measure this factor. Especially 
favorable attitudes were recorded on program characteristics deal- 
ing with the extent to which the ISP developed individual responsi- 
bility and the degree to which the program fulfilled students' 
immediate educational needs. 

Factor V : Degree of Challenge and Opportunity for Self- 
Expression, The five items which define this factor are concerned 
with the extent to which the independent study approach to 
learning is challenging and the opportunities that independent 
study provides for expressing ideas and feelings. High scores 
reflect positive attitudes in these areas. More than 70 percent 
of the students viewed the program as highly challenging and 
an equal proportion indicated that the ISP provided them with a 
great deal of freedom to express their own ideas and feelings. 
I nteraction with teachers was favorably evaluated by a majority 
of students; however, approxinr.ately one-third of the students 
, indicated that they would like greater opportunities to meet with 
consultants and experts on topics that they are pursuing through 
the ISP. 



Although a great deal of time and effort went into the construction of 
this instrument, student attitudes were considered to be a very important Key 
Feature in the Independent Study Program; and therefore, it was worthwhile 
to construct a valid and reliable instrument that could be used for continous 
program assessment over the years. Since the SATIS-Q yields mean scores and 
standard deviations, comparisions can be made to determine the effectiveness 
of programmatic efforts to improve attitudes on any or all of the five factors 
measured by the instrument. 

Additional sample instruments for evaluating both student and parent 
attitudes toward programs for the gifted are included in Appendix A. This 
appendix also includes an instrument that can be used to evaluate the attitude 
of non-program teachers toward the gifted programs. 

Another area related to attitude measurement is concerned with the feelings ' 
of non-program students toward participants in special programs. This issue 
emerged as a Key Feature in the evaluation of Project Gifted in East Providence, 
Rhode Island where gifted students spend part of their time each day in a 
regular classroom and part of their time in a special class. Using standard 
sociometric technique,* all students in the regular classroom were asked 
to indicate their first, second, and third choice with regard to whom they would 
liketo sjt near, work with, and play with. The results were tabulated on forms 
such as the one presented in Figure 16 so that means and standard deviations 



* See references by Gronlund (1959) and Gardner and Thompson (1958) in the Bibliography 
on Instrument Construction. 



Figure 16 



Name 


Sociometric Tabulation Form 

School 


Teacher 




Grade 


Sit Near 


Tally 


Number 


Weight 


Number 
Times Weight 


First Choice 






3 




Second Choice 






2 




Third Choice 






1 




Total Score 





Note: The same format should be used to record Work With and Play With data. 



could be computed for gifted and non-gifted students. The results of this inquiry 
are presented in Table 5. (Note the tabulation form and table show the 
results are included in the event that you may want to use this technique. The 
formula for computing the significance of difference between means can be 
found in most elementary statistics books.) As can be seen from this table, only 
three of the comparisons between gifted and non-gifted students were statistically 
significant , and in these three instances, the gifted students earned higher mean 
ratings than the comparison groups. Thus, one of the conclusions reached in this 
evaluation was that separating gifted students for special instructional purposes 
does not result in any detrimental effects so far as peer preferences are concerned 
and gifted students in the East Providence program appear to be equally accepted 
by the students in their regular classrooms. 
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Table 5 



Means, Standard Deviations, and Levels of 

Significance for Sociometric Status 
of Project Gifted Students, East Providence ' 



Groups 


Regular Class 


Gifted Students 






Level of 




mtan 


s.d. 


mean 


s.d. 


Difference 


t 


Significance 


Fourth Grade 
















Sit Near 


5.17 


4.67 


7.30 


6.34 


2.13 


1 .23 


n .s. 


Play With 


5.27 


4.35 


7.90 


6.23 


2.63 


1.61 


n.s. 


Work With 




•♦.01 


o an 


6.14 


5.05 


1 .59 




Total 


15.31 


12.68 


25.10 


17.57 


9.79 


2.07 


.05 


Fifth Grade 
















Sit Near 


5.27 


4.09 


6.64 


5.14 


1.37 


1.10 


n.s. 


Play With 


5.19 


3.93 


6.79 


5.06 


1.60 


1.33 


n.s. 


Work With 


5.30 


5.62 


6.50 


5.32 


1.20 


0.74 


n.s. 


Total 


15.76 


12.56 


19.93 


14.69 


4.17 


1.11 


n.s. 


Sixth Grade 
















Sit Near 


5.54 


3.91 


5.38 


4.65 


-.16 


0.13 


n.s. 


Play With 


5.20 


3.78 


4.23 


4.66 


-.97 


0.85 


n.s. 


Work With 


4.56 


3.23 


7.85 


6.36 


3.29 


2.98 


.05 


Total 


15.34 


9.78 


17.54 


14.60 


2.20 


0.71 


n.s. 


From: Renzulli 


, J. SuAn Eva/uation of Project Gifted. Storrs: University of Connecticut, 1973, 



b. Non-Test Meaures of Student Growth and Performance, Rating scales, 
checklists, logs, and anecdotal recording systenns are the types of non-test 
instrunnents that are nnost frequently used to evaluate student growth and performance. 
Several exannples of instrunnents that have been used for"these purposes are 
included in Appendix B. While nnost ratings of student perfornnance are designed 
to be connpleted by teachers, we note in Appendix B sonne exannples of parent 
and student scales that focus on a connbination of abilities, nnotivation, and 
general adjustnnent to participation in progranns for the gifted and talented. At 
this point we nnust raise a question about the appropriateness of sources of data. 
While it is certainly valid to ask parents about their childrens' enthusiasnn, 
attitudes toward various aspects of a special progrann, and activities that the parents 
can observe at home, we must be cautious when asking parents to rate student 
performance. Parents may simply not have the opportunity or knowledge to 
assess school performance adequately because they are somewhat removed from 
the center of the learning situation. When this is the case, their responses on 
performance rating scales may really be reflections of their youngsters' attitudes; 
and thus, so far as performance is concerned, we have used the wrong source of 
data. A good rule to follow in matching instruments with sources of data is 
simply never ask a person to second guess or answer out of ignorance. If respondents 
have not had numerous opportunities to observe the behaviors under investigation 
directly, then chances are they are not appropriate sources of data. 

Constructing instruments that can be used to assess growth and/or performance 
presents certain problems for the evaluator. Very often the items included in 
these instruments are too vague to pinpoint the specific areas where growth 
has occurred. For example, a statement that asks the teacher to rate 'The ability 
to use research skills," does not tell us exactly which skills the student has 
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mastered. An approach that focuses on relatively specific behaviors within the 
general area of research skills will help us to overcome the problem of vagueness. 
For example, we might ask the teacher to rate a youngster on the following 
behaviors: 

1 . The student used the following sections of (Scale) 
reference books to locate information: 

a. Table of Contents 

b. Index 

c. Appendix 

d. Bibliography 

e. Preface 

2. The student uses the following references 
appropriately in locating desired information: 

a. World Almanac ■ — : 

b. Readers' Guide to Periodical 

Literature 

c. Dictionary of American Biography 

d. Etc. 

3. When presented with information in the 
following forms, the student can interpret 
it and translate it into his own language: 

a. GraphS' 

b. Tables 

c. Topographical Maps 

d. Diagrams 

e. Flow Charts 

f. Etc. 

4. The student has demonstrated the following 
skills in analyzing and presenting data: 

a. Calculation of Means 

b. Calculation of standard deviations 

c. Preparation of tables 

d. Preparation of graphs 

e. Preparation of trend analysis statements . 

It is important to keep in mind that rating scales or checklists of this type 
must be developed in accordance with the specific content of particular 
instructional goals. The evaluator should careliilly review the curricular information 
that he obtained through Front-End Analysis, decide what type of instrument 
he will use, and then translate the information into an appropriate evaluation 
instrument. At this point the evaluator may want to work with teachers and 
curriculum coordinators to make certain that the instruments reflect accurate 
translations of the intended learning outcomes. It is also important to remember 
that a combination of instruments can be used to evaluate a particular component. 
For example, if a fairly valid test is available to assess some of the research skills 
discussed above, the evaluator may want to use this test in conjunction with a 
homemade rating scale that will cover skills not included on the test. 
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One of the areas that sometimes emerges as a Key Feature in the area of 
student growth has to do with self-concept development or other areas of psychologica 
adjustment. Generally, a great deal of psychometric know-how is necessary for 
instrument construction in this area and therefore it is recommended that evaluators 
consider already available instruments if they want to assess changes in self- 
concept. A book ent\t\ed Self-Concept Measurement: An Annotated Bibliography 
(Coller, 1971) is included in the Instrument Sourcebook section of the bibliography. 
We have, however, included one homemade self-esteem inventory in Appendix B 
to give you some idea of what is typically included in such instruments. 

Because of the many problems involved in evaluating the higher level 
objectives of programs for the gifted and talented, and in view of the great deal 
of individualization that usually characterizes such programs, alternative approaches 
to evaluation must be considered. One such approach, developed for the Alpha 
Project for Able Learners in the State of Washington, involves the use of an 
anecdotal recording system which is presented in Figure 17. This system is 
designed to provide a continous chronicle of student growth by {documenting 
several aspects of the learning process. The system focuses on specific "learning 
segments" for individual students, A learning segment is defined as any series of 
related activities directed toward the accomplishment of predetermined objectives.* 
A learning segment may require a day, a week or several months, and it is conceivable 
that a youngster may be working on several learning segments at one time. The 
system is mainly designed for individual documentation; however, if several 
students are working on a project together, and if their objectives are the same, 
the anecdotal record can very well be used for groups. There may, of course, be 
differences in the degree to which group members accomplish specific objectives 
and these differences should be recorded by individual rather than by learning 
segment. 

The instrument presented in Figure 17 provides a structured approach for ' 
recording evaluative information. Before discussing how the instrument can be 
used as an evaluation technique we should point out a few related features. First, 
the instrument encourages teachers and students to be a little more "objectives- 
oriented" in planning learning activities. Second, the students themselves can 
participate in planning and in so doing^ begin to learn the differences between 
various types of objectives. In places where versions of this instrument have 
been used students did much of the planning and record-keeping themselves. 
This involvement has caused students to become more responsible for their own 
education and has helped them to think more carefully about what they are 
studying and why they are studying it. Finally, in places where versions of the 
instrument have been used as a reporting system to parents, there have been 
very favorable reactions to questionnaires dealing with the degree to which parents 
have been informed about the program and the progress of their own children. 



* Note: 1 do not believe that a gifted youngster's total program must be rigidly planned and 
predetermined. All students need an opportunity to explore randomly, to play around, and 
to spend some time just being themselves. This type of eixploration may very well lead to 
the development of certain objectives, but I would like .to emphasize that I do not advocate 
the use of anecdotal recording systems such as the one in Figure 17 for every single thing 
that a child does in the gifted program. 
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When the instrument is used for evaluation purposes, the nnajor task 
is analyzing the content in ternns of the differences that characterize gifted 
progranns. The objectives for each learning segment are listed and classified 
according to their cognitive and affective levels. As the evaluator reviews 
the anecdotal records he should attempt to ascertain whether or not there is 
appropriate emphasis on higher level objectives. His judgment should be guided 
by the general goals or policy statement which underlie the program. Thus, 
for example, if the program places heavy emphasis on the development of creative 
thinking, the records should show a heavy concentration of objectives and 
activities that promote this type of mental process. The evaluator should study 
the objectives in relation to other information included in the record and attempt 
to determine if congruence exists between the Activities, Resources, and Products 
on one hand and the objectives on the other. 

The Areas of Study section will help to point out such things as inter- 
disciplinary studies or topics that do not fall into one of the traditional curricular 
areas. I n a similar fashion, the Activities, Resources, and Product sections will ^ 
help to reveal whether or not educational experiences and outcomes are generally 
different from those that occur in the regular school program. As information 
begins to accumulate, the evaluator must examine each section of the anecdotal 
record in terms of factors such as diversity, opportunity for self-expression, 
unusualness, and originality. He must constantly ask himself the question: 
"Would this student ordinarily have the opportunity to do these things in the 
regular school program?" 

Admittedly, this approach to performance evaluation places a great deal 
of reliance on (1 ) the accurate portrayal of information and (2) the experience 
and judgment of the evaluator. Nevertheless, it offers an alternative means for 
systematically describing and analyzing what ectually goes on in a program, and 
in this respect it may have certain advantages over tests. Boards of education 
and funding agencies frequently ask: "What is different about the program for 
the gifted?" and "What do students actually do in the gifted program that isn't 
being done in the regular classroom?" The answers to these questions reqdire 
descriptive information rather than test scores, but the descriptive information 
must be carefully analyzed in order to call attention to the differential nature 
of experiences. As the evaluator prepares a written analysis of anecdotal infor- 
mation, he should constantly focus attention on the quantity and quality of 
learning experiences and products which, in his judgment, represent relatively 
unique departures from the regular curriculum. 

Another approach that has been developed to provide information about 
what is going on in a gifted program is the Classroom Report (See Figure 18). 
Stake (1970) describes this report as follows: 

The purpose of the Classroom Report is to assist the teacher 
who desires to tell various people about her'classroom. We are 
acquainted with many teachers who want to tell parents or towns- 
people or administrators more about their student group and curriculum 
and teaching techniques than they can in informal conversations. Usually 
the writing chore is too great a burden. 

.. In developing this form, we have assumed that reporting on 
students as a group (rather than individually as you do with a report 
card) is a sensible thing for teachers to do. We believe that schools 
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have a responsibility to keep the community informed, and we 
believe that the work that goes into the preparation of a report 
like this helps the teacher put a number of important responsibilities 
into perspective. Furthermore-with increasing demands for evaluation 
and accountability— we feel that by making evaluation reports that 
are sensible to teachers and parents we can better resist irrelevant 
evaluation requif ements. 

A bit of advice for using the Classroom Report: (From the 
Manual for the Classroom Report). 

1 . Think of this as an opportunity to improve your com- 
munication with parents, administrators, townspeople-and even 
your mother. We're serious. Think of people you have been telling, 
or whom you would like to talk to, about what you are doing in 
your classroom. Then think of how you might use this form to get 
a good conversation started, or to improve an already-good 
conversation. 

2. Think who your readers will be. Parents, students, school 
board members, the Junior Chamber of Commerce? Will your 
Report be included in a project evaluation report? Different readers 
think that different information is important. Usually there won't 
be time to write up a different Classroom Report for different 
readers, but you probably can make one report relevant to a large 
audience. 

3. Use the form the way you want to. Change or add words 
as needed. Note that the report comes in three sheets (making 12 
pages) but some teachers use only one or two sheets. You can add 
your own sheets as inserts-but don't plan a gcand report that never 
gets done. 

4. Recognize that it is hard work to collect information— 
but that often the hardest-worked-for information is the best. . 
(Ask any newspaperman.) You have good information already. 
Some of it would be better information if you used a special 
instrument or procedure to collect it. This manual will suggest 
some possibilities. 
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Figure 18 
CLASSROOM REPORT 



Classroom Report 

ILLINOIS GIFTED PROGRAM 



ERIC 




STAFF 



STUDENTS 



STUDENT SELECTION PROCEDURES 



SPECIAL CLASSROOM FACILITIES 



SPECIAL BUDGET ALLOWANCES 
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The central idea of the Illinois Gifted Program is the he 
special talents of all children should be given specia' 
opportunity in school to develop. Each participating s^nool 
district decides how it will use the limited funds it gets. 
The following objectives have been set forth for the pro- 
gram in this classroom. 

CI ASSROOM OBJECTIVES: 



CLASSROOM EMPHASES: 

... to develop . . . 

Better ability to learn . . . . 
Problem solving abilities . . . 

Humanistic values 

Creative ideas 

Artistic performance . . . . 

Vocational skills 

Social criticism 

Self-awareness and esteem . . 
Content understandings . . . 
Sense of responsibility . . . . 
Respect for authorities . . . 



Emphasis given in this classroom 
Extra Usual None 



ACCELERATION or ENRICHMENT 

We attempt to have the students 

learn the same things that other students do but learn it faster. 

learn additional things that other students don't get much of a 

chance to learn in school. 
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MAJOR CLASS ACTIVITIES: 



USE OF SCHOOL TIME 

Individual Student Work 
Assigned reading 
Work on projects 
Exercises, tests 

Teacher Explanations 

Exchange of Ideas 
Recitation 
Inquiry 
Discussion 
Social, recreational 
Administrative matters 

The pie-chart above indicates approximately how time is spent in this classroom. 
The activities for any one day, of course, would not necessarily look like this. 

AN I LLUSTRATION OF STUDENT 
ACCOMPLISHMENT THIS PROJECT 
WOULD LIKE TO TAKE PARTIAL 
CREDIT FOR: 




88 



Scholastic ability, similar to what many people call IQ, was 
measured by the 



(test) 



on 



(date) 
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The bell-shaped curve at the left 
indicates the spread of scores of 
students this age across the nation. 
Each little box in the bar graph 
represents the score of one stu- 
dent in our classroom. 

The scholastic giftedness of our 
group is indicated by the cluster- 
ing of scores to the right of the 
arrow. 



STUDENT PROGRESS 

The following items are examples of the kinds of perform- 
ance this project emphasizes. Note the progress of the class 
during the period. 
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VIEWS OF THIS CLASSROOM 



Recently we tried to find out what 
others have been thinking about this 
classroom. Their views can give us 
a better idea about what is needing 
change and what is important to keep. 

Views were gathered from: 



Classroom's Features 

Best Needing 

Features Change 

Work on Thought-processes . . . 

Subject-matter coverage . . . 

Clarity of teaching . . . 

Student motivation . . . .__ 

Relevance to "real world" . . . 

Utility for later schoolwork . : . 

Pace of work scheduling . . . 

Workload . . . 

Chance for self-determination of work . . . . . . 

Facilities; materials . . . 

Class activities . . . 

Group atmosphere . . . 

Acceptance of individuals . . . 

Teacher competence . . . 

Student competence . . . 

Evaluations of students . . . 

Project self-evaiuation . . . 

Administrative support . . . 

Community support . . . 



Adapted from Stephen Lapan 
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STAFF EVALUATION OF THE CLASSROOM 



Several staff members associated with the 
Gifted Program have discussed the strengths 
and weaknesses of this classroom. Their 
views are summarized here. The marks'at the 
right indicate what they think needsThange 
and what is important to keep. 

Views were gathered from 

persons. Each was asked to name the best 
three things about the classroom and the three 
things most needing change. 



Classroom's Features 

Best Needing 

Features Change 

Work on Thought-processes . .* . 

Subject-matter coverage . . . 

Clarity of teaching . . . 

Student motivation . . . 

Revelance to "real world" . . . 

Utility for later schoolwork . : . . . 

Pace of work scheduling . . . 

Workload . . . 

Chance for self-determination of work . . . . . . 

Facilities; materials . . . 

Class activities . . . 

Group atmosphere . . . 

Acceptance of individuals . . . 

Teacher competence . . 

Student competence . . . 

Evaluations of students . , . 

Project self-evaluation . . . 

Administrative support . . . 

Community support . . . 



82 



Adapted from Stephen Lapan 
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Which Describe Our Classroom Best? 



□ Students move freely in 
and out of the room, taking 
much responsibility for the 
use of their time. 

□ When a student ex- 
plains something to the class 
the teacher is likely to praise 
him for his contribution. 

□ Students learn that 
growing up means "living 
by the rules"— and helping 
others to do likewise. 

□ Strong effort is made to 
get students to know the 
answer to questions such as 
those on standardized tests. 

□ Assignments are made 
clearly so that school time 
is used wisely and produc- » 
tively. 



□ When a student ex- 
plains something to the class 
the teacher is likely to invite 
him to discuss it further. 

O Laughter, small talk, 
an even occasional 
outburst are commcn in 
this classroom. 

□ Strong effort is made to 
get students to think 
throuo*' ,nd defend their 
own opinions. 

□ Most of the important 
work we do is in individual 
study and homework. 

□ Our test scores are less 
important to us than what a 
student does in working 
with others. 



n We make an effort to 
relate what we do in school 
to what adults do in their 
work. 

□ Students compete with 
each other. It is iq^rtant 
to them to get good marks. 

□ Most of the Important 
work we do is in group pro- 
jects in the classroom. 

□ What a student does on 
our tests is the best indi- 
cation of how well he is 
succeeding. ^ 

□ We emphasize that it 
is important for the stu- 
dents to decide (as a 
group) what is worth while. 

□ Students tutor each 
other. It is important to 
them that their friends 
learn too. 



Comment: 



Adapted from Joe Steele 
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THE ILLINOIS PLAN FOR GIFTED CHILDREN 

The legislation creating the Illinois Plan for Gifted Chil- 
dren states that the purpose of the legislation is to en- 
courage and assist in the developnnent of programs for 
students who are nnentally accelerated to the extent they 
can profit by special educational experiences. 

Because the Illinois Plan includes provisions for flexi- 
bility in identification and in educational experiences, 
encouragennent is given for providing special educational 
activities to the gifted child who is achieving at a high 
level, has exceptional creative ability, and/or has po- 
tential to be outstanding in any one of the nnany traits 
included in giftedness even though the student nnay not 
have as yet developed those talents. 

The support and encouragennent given to schools to 
identify various kinds of high-ability students, and to de- 
velop special progranns for individuals, increases the re- 
trieval of talent at various socioecononnic levels as well as 
at all educational levels in the public schools of Illinois. 



STATE STAFF 

Superintendent 

of Public Instruction . . Ray Page 

Deputy Superintendent Verne E. Crackel 

Assistant Superintendent, Division of Special 

Education Services , Robert E. Cain 

Director, Departnnent of Progrann Developnnent 

for Gifted Children Herbert Baker 

Assistant Directors,^" ^ 

Departnnent of Progrann Developnnent for 

Gifted Children Robert Hardy, June Cox 



This report was prepared by the Center for 
Instructional Research and Curriculum Evalu- 
ation, University of Illinois at Urbana-Cham- 
palgn, 1969. 
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c. Inservice Training. One of the areas that frequently emerges as a 
Key Feature in programs for the gifted is the value and effectiveness of in- 
service training or staff development activities. These activities are usually 
evaluated by means of questionnaires and rating scales that are administered 
at the end of individual training sessions and at the conclusion of workshops 
or institutes. Sample questionnaires that have been developed for these purposes 
are included in Appendix C. 

Two considerations should be kept in mind when construrxjpg irstruments 
to evaluate inservice training. First, the instruments should focus on (1) the 
importance or relevancy of the material that was covered in training, and (2) * 
the effectiveness with which specific topics or skills were taught. Negative 
reactions to inservice training are often a function of the appropriateness of 
the content rather than how well it was covered, and therefore, it is important 
for the evaluator to find out wl/- selected the topic(s) and trainer(s) for the 
workshop, why they were selected, and if the participants had any input in 
the planning. As far as effectiveness is concerned, there is little value in obtain- 
ing feedback unless we can identify specifically which areas were or were not 
covered adequately. Thus, an attempt should be made in instrument construction 
to break down training activities into their component parts so that we avoid 
the vagueness and uncertainty that frequently results when questionnaire 
items are too general. 




miJ>R^Bh Bori 'u)nT kno^ If ouie, euenr wcucb Bui 



This leads us to the second major consideration in evaluating inservice. 
training— how feedback information will be used. In far too many cases, 
workshops, institutes, and especially visiting speakers are one shot deals that 
may be good or bad, and there is little anyone can do about it (except perhaps 
to walk out) once a particular session or speaker has gotten underway. In order 
for the evaluation of training to be of any value whatsoever, the information 
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obtained from questionnaires or rating scales must lead to certain conclusions, 
actions, and perhaps follow-up evaluation activities. For example, if teachers 
evaluate a workshop on creativity training as being very effective in providing 
them with the skills necessary to teach. creative writing, but ineffective in the 
area of creative dramatics, then we may want to recommend certain actions. 
If creative dramatics is an important part of our program, perhaps we will want 
to redefine our expectations for this area of training and seek another person 
to conduct a workshop. A report of the previous workshop and a list of 
expectations should be given to the trainer prior to the workshop so that he 
or she will be able to focus the training appropriately. Another action may be 
to restrict the original trainer's involvement in future workshops to creative 
writing; or, if we are stuck with this person for a long period of time, to provide 
him with the feedback and expectations so that he will improve his presentation 
in creative dramatics.* Follow-up evaluation activities might focus on student 
products and/or classroom processes that we expect will take place as a result 
of the training. 

The main point here is very simple. The evaluation of inservice training 
should have a future programmatic orientation; otherwise there is little value 
in going to the trouble of gathering such (3lta. All too often, we waste people's 
valuable time by asking them to fill out questionnaires that do not have a 
particular purpose so far as program improvement is concerned. An analysis 
of these purposes will help persons responsible for planning in service training 
to look more carefully and systematically at their overall training program. 

d. Program Operatior). A final area of consideration in instrument construction 
deals with Key Features that relate to the ways in which programs are operated. 
Prime Interest Groups such as teachers, parents, and board of education members 
may, Tor example, be interested in how the program is managed, how 
information is disseminated, and how teachers react to certain program 
activities. The Management Evaluation Form included in Appendix D was 
designed to evaluate the following four characteristics of a program director: 
Communication, Management Style, Planning, and Creativity. These character- 
istics were determined through Front-End Analysis techniques, and since 
several persons expressed concerns about how effectively the project director 
related to various groups, the instrument was administered to all of the groups 
listed on the last page of the form. This application helps us to illustrate an 
important point in instrument construction. Since various group members 
interact with the project director in different ways, some of the questions on 
the form are clearly inappropriate for certain respondents. For this reason, a 
"No Opportunity to Judge" or "Not Applicable" column was included in the 
rating scale. This approach avoids the problem of forcing people to respond to 
items which may be irrelevant to them. The results of this instrument and the 
feedback to the director were classified according to the various categories of 
respondents (Sources of Data, and in this way the director could use the infor- 
mation for improving his interaction with each respective group. 



* I believe that trainers end speakers should always be provided with feedback even If they 
are not scheduled for additional work in a particular school. If the feedback information 
helps to improve the presentation then it may benefit other persons that the speaker will 
address. In this case, evaluation will have made a contribution to the movement of gifted 
education at large. 
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Appendix D also includes some sample follow-up questionnaires that were 
used to determine the relatively long range impact of programs for the gifted and 
talented. In most cases, these questionnaires deal with several types of information 
and they could very well be categorized under a variety of headings. They are 
included here as examples of the kinds of information that typically is sought 
in follow-up studies. We have also included a sample rating scale that was developed 
to evaluate a particular aspect of program operation. The instrument entitled 
J'The Evaluation of Identification Procedures by Selection Committees" was 
designed to determine the usefulness of several types of information that are 
used in a case study approach to student identification. The results are intended 
to help streamline the identification system by eliminating information that is 
of limited value in decision making. 

3. Summary, Instrument selection and development is one of the most 
difficult and time consuming tasks of the evaluator. Three major concerns 
should guide his work in this area. The first is an accurate translation of Front- . 
End Analysis information into Key Features. The second is establishing appropriate 
relationships between Key Features and the instruments that will be used in 
data gathering. This step includes careful consideration of the psychometric 
qualities of the instruments and identification of appropriate sources of data. 
Finally, the evaluator must be concerned with economy and parsimony as they 
relate to the entire data gathering operation. Too many tests, questionnaires, 
interviews, etc. can have a negative influence.on the attitudes of people toward 
evaluation; and this, in turn, may affect the ways in which they respond to 
the evaluator and his tools. 

C. Step III. Data Collection and Analysis 

1 . Data Collection, We have used the term "data" to refer to any type 
of information that is collected for evaluation purposes. Some data will undoubtedly 
consist of test scores and statistical summaries, but the term is used here in a 
broader context to include the kinds of information that are derived from samples 
of students' work and descriptive material such as the information that might 
be collected on anecdotal recording systems (See Figure 17). One of the practical 
concerns in data collection is appropriate timing. After the evaluator has 
identified Key Features, Sources of Data, and instruments necessary for obtain- 
ing the data, he must then decide when the data will be gathered and who will 
gather them. These may appear to be simple tasks, but without careful planning 
important information can be lost or the value of very good instruments can be 
diminished. Suppose, for example, that an evaluator decides to use Xhe Class 
Activity Questionnaire (CAQ) to determine if any changes have taken place 
'over the course of a school year in levels of thinking and classroom climate. 
In this situation collecting comparative information is necessary and therefore 
the evaluator should make plans to gather the initial data clearly in the school 
year, but not before teachers and students have settled down to what is a fairly 
representative classroom environment. Collecting the data during the first week 
of school may present a highly distorted picture of classroom conditions. If, on 
the other hand, the evaluator decides half way through the year that he would 
like to do a comparative analysis of CAQ results it may be too late to document . 
the classroom conditions that existed during the early stages of the program. 
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There are very few evaluators who have not said to themselves at one time or 
another: "Too bad I didn't get some information on (a particular skill or 
activity) back when the program was just getting started." Although changes 
in an evaluation design frequently take place after the evaluation has gotten 
underway, there is no substitute for careful and comprehensive planning before 
the program gets started. Even "little things" like school vacations, final exami- 
nation schedules, or special events can play havoc with the timing of data 
collection, and therefore, it is important for the evaluator to develop a time 
line that takes into account both psychometric and practical concerns. Many 
techniques and systems have been developed to chart the sequential planning 
and work-breakdown structure of projects, but perhaps the best known approach 
is the Program Evaluation and Review Technique (PERT) which was designed 
to aid in the planning of many diverse activities (e.g., missile development, 
Broadway plays, research projects). Persons who have been responsible for large 
a- id complicated evaluations have found the following basic reference to be 
quite useful: 



BASIC REFERENCE on Sequential Planning 



The PERT system can get very complicated and therefore I would 
suggest that you use this reference to get ideas rather than to 
follow it slavishly. A practical hint in sequential planriing is to 
have a school calendar laid out on a single page so that you can 
view the entire year at a glance. 

Cook, Desmond L. Program Evaluation and Review 

Technique: Applications in Education. Washington, D.C.: 
U.S. Government Printing Office, 1966. (Price: 45 cents) 



Timing is also important in terms of how often information is gathered and 
how much time is required to obtain the information. If parents, for example, 
receive too many questionnaires, or if the questionnaires are long and repetitious, 
they may respond to them in a haphazard fashion. Asking the same questions 
on rating scales ano^ interviews might be a waste of the respondent's time 
(unless the evaluator has a particular purpose in mind such as cross-checking), 
and therefore, careful consideration should be given to the time-dimension of 
all data collection activities. It is a good idea to field test each instrument in 
order to obtain an estimate of the amount of time that it will require. Trying 
to squeeze a one-hour instrument into a forty minute period can result in the 
loss of valuable information and it may produce both frustration and hostility 
in the respondents. 

Another concern in data collection relates to the issue of scaling. This is 
an extremely technical aspect of psychometrics that has been hotly debated by 
behavioral scientists for years. Although scaling is a topic that is of primary 
concern to instrument developers, the precision with which evaluative data can 
be analyzed is usually a function of the type of scale employed in data collection. 
Decisions regarding whether to use a "Yes — No" scale, a 5-point scale (e.g., 
Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree), or some other- 
scale are probably best left to the measurement specialist, but the field 
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evaluator may want to review some of the advantages and disadvantages of 
various scaling techniques. A number of the books in the bibliography on 
Instrument Construction deal with this topic and one book (Maranell, 1974) 
is devoted entirely to problems and issues related to different scaling procedures. 

Before leaving this topic we should point out a few obvious, but very 
practical hints in data collection: 

a. Research has shown that stamped self-addressed return 
envelopes produce higher return rates for mail question- 
naires 

b. Mail questionnaires should always have a letter of intro- 
duction that includes a statement indicating that program 
personnel are knowledgeable and supportive of the 
evaluation. The letter should emphasize the positive purpose 
of the evaluation (program improvement) and indicate how 
the results of the evaluation can be obtained when they are 
available 

c. Letters and questionnaires that are sent to parents should be 
meticulously screened for errors in spelling, grammar, and 
facts about the program. The evaluator loses credibility in 
the eyes of some respondents when sloppy work is sent out 

d. Questionnaires administered to students in small groups 
result in less goofing off than in large group settings 

e. Whenever possible attitude questionnaires should be com- 
pleted anonymously 

f. Students sometimes feel intimidated even when completing 
anonymous questionnaires, and therefore, a non-program 
person should administer any instrument that asks for frank 
and honest opinions about the program or program 
personnel 

2. Data Analysis, Data analysis involves the breaking down of information 
into its component parts so that we can make sense out of it. The way in which 
information can be broken down should be planned in the earliest stages of an 
evaluation so that we do not end up with data that is "unanalyzable." 
Generally speaking, two methods of data analysis are open to the evaluator 
once he has gathered information related to each Key Feature. These methods 
are logical analysis and statistical analysis. 

. a. Logical Analysis. Descriptive information such as the type that rright 
result from anecdotal recording systems (See Figures 17, 18) must be logically 
analyzed in much the same way that a historian attempts to reach conclusions 
from the events about which he reads and observes. Information must be 
categorized according to some common characteristic and an attempt should be 
made to discover patterns, trends, or discrepancies that exist within each clearly 
discernable category. 

Perhaps a very simple hypothetical example will help to illustrate how 
this process can be carried out. Let us assume that one of the major concerns 
of several Prime Interest Groups is: "How does the program for the Gifted 
differ qualitatively from the regular school program?" Let us further assume 
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that the evaluator has targeted ten gifted students who are participating in a 
special language arts senninar to serve as sources of data. The students and 
teachers have been asked to connplete anecdotal records like the one in Figure 
1 7.. Finally, let us assunne that we have access to infornnation about the regular 
language arts classes in which these youngsters would have been enrolled had 
they not been in the special progrann. This infornnation might consist of lesson 
plans, curriculunn guides, and sannples of student's assignments. After a one 
month time sample has been collected, the evaluator classifies information 
from each section of the record. In the Resources section he may note, for 
example, a trend in the seminar toward using more advanced level books and 
a wider variety of reference materials. The Activities section may reveal that 
the Seminar devotes a great deal of time to debates and student-led discussions, 
whereas the regular program emphasizes drill and skills development activities. 
After classifying items in the Product section according to whether or not they 
are convergent or divergent, the evaluator notes that products from the gifted 
program are much more divergent than products prepared by the same students 
in their regular language arts class. Armed with this information, the evaluator 
can make a case for the qualitative differences of the gifted program, and if 
called upon he can display the records and samples of students' products to 
support his assertion. 

Records of this type only provide facts bnd the information will be 
valueless without painstaking efforts to highlight the patterns, the implications, 
and the logic of the material that has been gathered. This approach helps the 
evaluator to make sense out of unrefined and unclassified descriptive information 
and thus it represents a more sophisticated way of analyzing non-statistical 
data. While the approach might sound complicated or time consuming, it is 
not unlike the type of analyzing that evaluators have been doing for years 
when they deal with interview information and the results of open-ended 
questionnairos. 

b. Statistical Analysis. The second method of data analysis is the well- 
known statistical approach. We will only deal briefly with this topic because 
it is an area that generally requires the assistance of a specialist or advanced 
levels of training that are beyond the scope of the Guidebook. Although we 
will point out a few of the major considerations that the evaluator should be 
concerned with when he is analyzing statistical information, it is recommended 
that further reading be done in this area by persons who are either conducting 
highly statistical evaluation or who must communicate with statisticians. The 
bibliography on Measurement, Research Design and Statistics includes 
references on statistical analysis that have been carefully selected for the clarity 
with which they present this very complicc'ed aspect of evaluation [See, for 
example, Popham and Sirotnik (1973) and Garrett (1968).] 

The evaluator generally uses statistics for two primary purposes. Descriptive 
statistics are used to summarize large sets of numerical information so that we 
can quickly learn the status or characteristics of groups on particular variables. 
Means, standard deviations, medians, percentiles, and grade-point averages are 
examples of descriptive statistics that are frequently found in evaluation reports. 
They provide a vehicle for further analysis but do not, in and of themselves, make 
judgments or render opinions. And while we may get "a .feeling" for conclusions 
by simply inspecting a set of scores, more sophisticated techniques must be used 
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in order to interpret accurately the meaning of descriptive statistics. Thus, the 
second purpose of statistics is to help us draw better inferences fronn the nunnerical 
material that we have gathered. The evaluator generally uses inferential statistics 
to determine if differences between scores are significant. For example, if a 10 
point gain is observed in the mean scores on a post-test of creativity, inferential 
statistics can. tell us the probability of whether or not this is a "real" gain or a gain 
that can be attributed to chance alone. I n the jargon of the trade, we say that 
such a gain is or is not "statistically significant." Tests of statistical significance 
include t-tests, analysis of variance and covariance, multiple regression and many 
other less popular techniques. 

Although this is an oversimplification of the differe^^.^ between descriptive 
and inferential statistics (Indeed, my statistical friends woujd boil me in oil!), 
we will use it as the basis for pointing out some concerns with which the evaluator 
typically is faced. The first concern is selecting the appropriate descriptive 
statistic for further analysis. A simple example: if the evaluator is interested in 
comparing mean differences between groups he will be unable to do so if he ends 
up with percentile scores which cannot be averaged because of uncomparable 
differences between score points. Thus, the evaluator must think ahead and 
organize his work carefully so that he can end up with information in the proper 
form for data analysis. It is a good idea to seek the assistance of a statistician 
at the beginning of an evaluation study rather than after the data have been 
gathered. Elaborate statistics are of little value if basic underlying assumptions 
in design have not been met, and therefore appropriateness is an absoutely 
essential concern in statistical analysis. 

A second concern in statistical analysis is the audience to whom the evaluator 
is reporting. Highly complicated statistical reports oftentimes serve limited 
purposes simply because the audience does not understand the statistical 
mystique that surrounds answers to questions which they have raised. It is a 
good idea for the evaluator to estimate the audience's level of sophistication 
during Front-End Analysis and attempt to report his findings accordingly. 
Many practitioners who simply don't trust highly complicated statistics may 
have a negative attitude toward an entire evaluation report without even knowing 
the content of the report. When it is necessary to use complicated statistical 
concepts, the evaluator should take the time to explain their meaning in his 
written and oral reports. 

D. Step IV: Preparing Evaluation Reports 

1 . Organizing the Report This section will deal mainly with the 
preparation of final reports; however, interim reports and briefs should follow 
a similar format. Interim reports usually deal with specific components of Key 
Features and in formative evaluation studies they are intended to provide in- 
process feedback so that appropriate modifications can be made as the program 
progresses. The information presented in interim reports can be included in 
final (Summative) reports; and in a carefully planned evaluation study, the 
evaluator can actually draw a good deal of the content for his final report from 
the interim reports. A good time-saving device for the evaluator is to view each 
interim report or brief as a component part of his final report. 

A very important consideration to keep in mind when preparing all reports 
is that the evaluator has a responsibility to discuss both the positive and negative 
aspects of his findings. All too often the reports of programs which d^re generally 
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excellent dwell on the few areas that are in need of improvement. This practice 
sometimes gives the reader a very distorted perception of the program. 

Let us assume the evaluator has collected all of the information he will 
need to file a final report. At this point he should review the major concerns 
of each Prime Interest Group and view each concern as a question the report 
will attempt to answer. The report should begin with-an introductory chapter 
that contains a description of the program and an overview of the evaluation 
design. The overview should point out: 

a. Why the evaluation was conducted 

b. What question the evaluation will answer 

c. The rationale for using particular methods and techniques 

d. Problems that were encountered and any modifications 
that had to be made in the original evaluation design 

e. When and how the data were collected 

f. A brief forward-statement that describes the content of each 
chapter, section, and appendix that will follow 

The overview is very important for a number of reasons. Very few people 
sit down and read an evaluation report from cover-to-cover; and thus, the 
overview helps to identify sections that will be of special interest to each Prime 
Interest Group. The rationale and problems sections (items c and d above) will 
let the reader know why the evaluator did and did not do certain things, and 
the logic underlying the evaluator's particular approach. It is at this point that 
the special problems of evaluating programs for the gifted and talented might 
be summarized (see Chapter Two). If the overview is carefully. written it will 
help to "protecf'the evaluator from questions such as "Why didn't you use 
more tests?" or "Why didn't you use a control group?" 

Each chapter thot follows the introductory chapter should be organized 
around one Key Feature. An overview of tho chapter can be conveniently 
provided by including the Data Collection and Analysis Guide for the Key 
Feature under consideration. Within each chapter, the subsections should deal 
with the Activities Being Evaluated which are listed in.the first column of the 
Guide (see Figure 15). For e.ach Activity Being Evaluated, the report should 
discuss the information contained in the remaining columns of the Guide. The 
Source of Data or sample population should be described in terms of its size 
and characteristics (age, sex, grade, etc.). The Data CoilectionlMethod should 
briefly describe the instruments or techniques that were used, any information 
that helps to support the reliability and validity of the instruments, and how 
the instruments were selected or developed. Reference should be made to 
sample copies of the instruments that are included in the appendix and 
appropriate citations should be included whenever material or supportive 
information is taken from other sources. 

After the techniques for data analysis and the time schedule have been 
described, the evaluator should present the results as they relate to each Activity 
Being Evaluated. It is at this point that he must decide which methods of data 
presentation will result in the clearest interpretation of findings. Numerical 
information should be summarised in statistical tables and graphs should be 
used to highlight growth patterns or trends that may not be readily discernable 
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from the tables. Most measurement and statistics books will provide you with 
examples of the format to follow in constructing tables and graphs. As has 
been indicated earlier, it is important to consider the level of sophistication of 
the audience when interpreting findings; and in cases where persons with different 
backgrounds will be receiving the report it may.be necessary to include both 
technical and non-technical presentations. All tables and graphs should be 
carefully explained in narrative sections and each narrative should end with the 
conclusions that the evaluator has drawn from the data. If information is open 
to various conclusions and interpretations these shou^d^Wlso be pointed out. 
In fact, evaluative data will frequently include conflicting information or 
information that does not lead to clear-cut conclusions. In these cases, the 
evaluator should simply present both sides of the issue. A valuable but often 
neglected form of reporting conflicting points of view is simply to present 
examples of verbatum comments that have been obtained from interviews and 
questionnaires. The comments should be grouped together according to similarities 
and differences or presented in a point/counterpoint fashion. This information 
will help to emphasize areas of agreement and disagreement and at the same 
time, make the reader aware of the reason why firm conclusions cannof be drawn. 

Each chapter should end with a brief summary that highlights the major 
conclusions that are related to the Activities Being Evaluated. These conclusions 
will serve as the bases for making recommendations. 

The final chapter of the report should contain a summary of the entire 
evaluation and recommendations that seem to be warranted by the findings. In 
some cases the summary information in this chapter will be repetitious of 
earlier sections; however, it should be kept in mind that many people will only 
read the final chapter. (A good idea is to write the first and last chapters in 
such a way that they can be put together to provide a brief but thorough 
picture of the entire evaluation. This shorter document is often useful for 
disseminating information to audiences other thsr\ key decision-making or 
funding agencies.) ^' 

Following a general overview, each section of the final chapter should be • 
organized around a single Key Feature. The results and conclusions should be 
summarized and the recommendations should be listed. We have already 
discussed (see Chapter One) the importance of realistic recommentations; 
however, we should emphasize that the evaluator has a responsibility to present 
alternative recommendations that range from what he considers to be ideal 
courses of action to actions that are measured against available funds and 
resources. A time-perspective should also be taken into account when making 
recommendations. Thus, for example, if evaluative findings lead to the con- 
clusion that several new areas of study or grade levels should be added to a 
particular program, the evaluator may want to suggest that these areas be 
systematically added over a two or three year period. This suggestion may be . 
much more realistic than simply recommending that the changes be made en 
masse. 

The report should end with a general summary that highlights both the 
strengths of the program and those areas that are in need of improvement. 
After the report has been submitted, the evaluator should plan to meet with 
representatives of Prime Interest Groups to discuss his findings and to answer 
any questions that they may have. It is a good idea to be prepared to present 
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any backup data or information that will help to substantiate conclusions and 
recommendations. 

2. Technical Aspects of Report Writing. Although all written work 
should be of the highest possible quality, there are special reasons why it is 
doubly important for the evaluator to produce accurate, well-organized, and 
attrative reports. Evaluation is a very touchy business and when the eyiauator 
comes up with findings that are viewed as a threat by individuals or groups, 
they may try to raise questions about the evaluator's credibility. The easiest 
way to do this to to begin by criticizing the technical aspects of the evaluation 
report. Errors in spelling, poor grammar, and obviously incorrect facts or 
statistics immedicately put the evaluator on the defensive; and thus, he is 
"softened-up" for criticisms that may deal with his more substantive conclusions 
and recommendations. And once the evaluator is forced to admit error and 
inaccuracies on small things that could have been avoided, he may have set a 
pattern for backing-off on some of his more important conclusions. For this 
reason, evaluation reports should be as technically perfect as possible. It takes 
only a little extra effort to have reports edited, proofread, and reviewed for 
accuracy, but the protection that these steps give to the evaluator's credibility 
are well worth the effort. 
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CHAPTER FOUR: Selecting an Evaluator 
and Negotiating an Evaluation Contract 

This brief chapter is primarily intended for persons who will be responsible 
for selecting external evaluators and negotiating with thenn on a contractual 
basis. We have already discussed some of the issues that are related to the technical 
competencies of the evaluator; and therefore our concern here will be with 
the practical aspects of selection and with the steps that might be followed in 
moving from the needs of persons who are hiring evaluators to evaluation proposals 
and contracts. 

The primary concern in identifying and selecting a qualified evaluator is 
that he or she possess the ability to prepare a comprehensive evaluation design. 
Although a careful design will not guarantee a successful evaluation, it is a 
prerequisite to success and therefore the evaluator's most important qualifications 
should be in the areas of planning and management. Because of the rel? .ively 
unique characteristics that frequently are associated with programs for the gifted 
and talented, it would also be a good idea to seek out evaluators who have had 
some experience in this area. Beyond these general qualifications, the type of 
program under consideration should serve as a guide in identifying the specific 
technical competencies that are necessary for carrying out particular kinds of 
evaluation studies. If, for example, a great deal of value is placed on test-related 
information by persons who will receive an evaluation report, then it seems 
quite obvious that we should seek out an evaluator with a strong background 
in measurement and statistics. Similarly, if a program deals mainly with a non- 
traditional area in which very few valid and reliable instruments are available, 
a major competency that we should look for in an evaluator is the ability to 
design and construct appropriate instruments and other data gathering techniques. 
Curriculum expertise is of ten necessary for programs that focus on particular 
areas of study; and an evaluation that depends heavily on interview information 
will require a person who relates well to people in face-to-face situations. When 
many diverse competencies are required in an evaluation study, it may be 
necessary to consider a team approach rather than a single evaluator. The chief 
evaluator or the person with whom a contract is negotiated may not necessarily 
possess all of the required technical skills; but, it is his or her responsibility to 
insure that appropriate specialists are involved. 

The best criterion for selecting an evaluator is past performance. Although 
a person's credentials will tell us a great deal about the specific areas of competency 
that he or she possesses, actual examples of previous work are usually the best 
indicators of whether or not a prospective evaluator can deliver the kind of 
product that is required in a particular evaluative situation. A good idea in 
screening prospective evaluators is to ask them to submit examples of previous 
reports ancy the names of persons for whom they have carried out evaluation 
studies. In reviewing past accomplishments and letters of recommendation, it 
is important to estimate both the technical competencies of the evaluator and 
his availability and responsiveness to the programs for which he has worked. 
Quite obviously, all factors must be weighed in relation to one another. For 
example, if an evaluator's report is extremely critical of a certain program, then 
we must consider the program director's opinion of the evaluator in light of the 
criticism and the director's possible reaction to it. The task here is clearly one 
of determining if the evaluator has done a conscientious job and if the criticism 
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in our hypothetical example is supported by the information contained in the 
report. 

An important consideration in selecting an evaluator is the orientation or 
point-of-view that the evaluator brings to his work. Persons with a strong research 
orientation, for example, might be likely prospects for an evaluation that is 
primarily summative and affords many opportunities to collect statistical data, 
however this orientation may present problems if we are mainly interested in a 
formative evaluation design. Similarly, evaluators who follow a fairly strict 
behavioral objectives approach may have difficulty working with programs in 
which outcomes cannot be precisely stated in observable and measurable terms. 
In short, problems will invariably arise if evaluators and program personnel 
have different conceptions of what is to be evaluated^and how the evaluation 
should be carried out. The best way to avoid such problems is to draw up a 
set of specifications that accurately describes each component of a program 
and the related evaluation questions that program personnel would like to have 
answered. This information is frequently contained in . program proposals and 
should include: 

1 . The audiences that will be served by the evaluation 

2. The types of decisions for which the evaluation will be 
used 

3. The philosophy underlying the program 

4. The types of data that will be available and any conclusions 
that may b& placed on data gathering activity 

5. The anticipated outcomes of the program 

6. The amount of money available for evaluation 

7. A time-line for reporting evaluation findings. 

Based on this information, prospective evaluators should submit proposals 
on a competitive basis and the person whose approach seems most likely to answer 
the questions should be invited to negotiate an evaluation contract. Because of 
differences in evaluation situations it is difficult to specify an exact format for 
proposals; however, the following information should be included in most types 
of proposals: 

1 . The general evaluation design 

2. The types of instruments that will be used to evaluate each 
area of concern and the rationale for selecting or developing 
instruments 

3. The data collection procedures and a time schedule of data 
collection activities 

4. The procedures for analyzing data relative to each instrument 
and technique 

5. The methods for reporting evaluation finding^ and a time- 
schedule for the submission of interim and final reports. 

Many of the problems between evaluators and persons sponsoring evaluations 
can be avoided if the specific responsibilities of each party are carefully spelled 
out beforehand. In Appendix E we have presented a sample evaluation contract 
that attempts to point out areas of responsibility that typically are included in 
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formal evaluation agreements. Differences in local policies and regulations 
will do doubt require variations in contracts, but it is a good idea for all 
agreements to contain a "Statement of Responsibilities" such as the one that 
is included in Appendix E. It is also a good idea for the agreement to specify 
the information that the evaluator will be allowed to gather and the areas where 
restrictions will be placed on his activities. 

The evaluation proposal and contract are fequently combined into a single 
statement that can be used to guide the procedures for an entire evaluation. 
Together they represent a legal and programmatic document that can serve as 
a checklist for determining the degree to which the evaluator has fulfilled his 
responsibilities. 

By way of summary, the selection of an evaluator and the agreements 
that must be reached between the evaluator and sponsor are considerations that 
can be extremely influential in determining both the type and quality of an 
evaluation study. There are very few persons writing in the field of evaluation 
who do not stress the obvious, but often overlooked fact that the types of 
information the evaluator decides to collect and the ways in which he interprets 
information will influence his findings, conclusions, and recommendations. 
Deciding and interpreting are human-judgment activities, like the three umpires 
in the following cartoon, each evaluator may perceive his responsibility in a 
somewhat different way. 
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personality and emotional characteristics; children's perceptions of environment; 
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self-concept; characteristics of environment; motor skills, brain injury, sensory perception; 
physical attributes; attitudes and interests not otherwise classified; social behavior and 
measures that do not fit any of the above categories. The information provided for each 
measure includes: name of measure, author, age of children, variable, type of measure, 
source from which the measure may be obtained, description of measure, reliability and 
validity, and bibliography. 

Johnson, T., & Hess, R.J. Tests in the Arts. St. Ann, Missouri : Central Midwestern 
Regional Educational Laboratory, 1970. 

Index of instruments that measure abilities in the following areas of the Arts; art, dance, 
drama, films, literature, music and creativity. The following information is provided on 
each entry: title, author, key words, sources, category and resume (item, scoring, 
validity, reliability). A taxonomy for each domain is included which classifies instruments 
according to what they measure and what specific method is used to measure them. 

Robinson, J. P., Athanasiou, R., & Head, K. B. Measures of Occupat/onai Attitudes and 
Occupationai Characteristics. Ann Arbor: University of Michigan Institute of Social 
Research, 1969. 

Review and evaluation of 77 scales measuring variables related to a person's occupation. 
Scales are grouped into the following categories: general job satisfaction scales; job 
satisfaction with specific job features; concepts related to job satisfaction; occupational 
values; l.-adership styles, other work-relevant attitudes; vocational interest measures and 
occupational status measures. Other topics included in this volume are: research on work 
and workers in the U. S.; a summary of survey research evidence on occupational norms 
and differences in job satisfaction; a review of literature on job attitudes and occupational 
performance; occupational situs; steps for measuring social mobility and occupational 
similarity. 

Robinson, J. P., & Shaver, P. R. l\/leasu res of Social Psychological Attitudes:. Ann Arbor: 
University of Michigan, Institute of Social Research, 1970. 

Listing and evaluation of 106 measures of social attitude variables. The scales that are 
included were grouped into the following categories: alienation and anomie; authori- 
tarianism, dogmatism and related measures; other socio-political attitudes; values; general 
attitude toward people; religious attitudes and methodological scales. The criteria for 
scale construction which were used to evaluate the scales were: (1 ) item construction 
criteria, (2) response-set criteria and (3) psychometric criteria, v 

Shaw, M., & Wright, J. M. Scales for the Measurement of Attitudes. New York* McGraw- 
Hill, 1971. 

Compilation of scalts that measure attitudes toward social practices, social issues and 
problems, international issues, abstract concepts, political and religious systems, ethnic 
and national groups, significant others and social institutions. Chapters are included that 
consider the general nature of attitudes and methods of measurement and evaluation and 
suggestions for improvement of scales. 

Simon, A., & Boyer, E. G. (Eds.) Mirrors for Behavior III: An Anthology of Observation 
Instruments. Wyncote, Pennslyvania: Communication Materials Center, 1974. 
Compilation of 99 observation instruments, most of which deal with "learning" environ- 
ments. The systems were grouped into seven major classes: affective, cognitive, psycho- 
motor, activity, content, sociological structure and physical environment. Chapters are 
included which discuss the technology of observation instruments and present and 
future applications of observation instruments. Also included is an extensive bibliography 
which references research reports, user manuals, theoretical papers and reports of training 
projects. . ?. 
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Test Collection Bulletin, Princeton, New Jersey: Educational Testing Service. 

This is a quarterly publication describing standardized tests and other measurement devices. 
It Includes publishers' catalogs, information on testing programs and scoring services, test 
reviews, reference materials on measurement and evaluation and the address of the publisher 
or organization where each test can be secured. 

Walker, D. K. Socioemotional Measures for Preschool and Kindergarten Children, San Francisco, 
California: Jossey-Bass, 1973. 

Gu'de to 143 socioemotional measures available for use with child'en aged three to six years. 
The scales are classified into six area: Attitudes; General personality and emotional adjust- 
ment; Interests or preferences; Personality or behavior traits; Self-concept; and Social skills 
or competencies. Descriptions of each measure include: title and date of publication or 
copyright; author; appropriate age range? measurement techni jjue, sources in which the 
measure is described; addresses from which the measure can be obtained; a description of 
the instrument; norms available and validity and reliability evidence. 
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APPENDIX A 
Sample Instruments: Attitude Assessment 



Title 

1 . Student Attitude Toward I nde- 
pendent Study Questionnaire 

2. Staff Sennantic Differential 

3. Parent Questionnaire 

4. Mentally Gifted Minor Progrann 

5. Instrunnent for Teachers 

6. Parent Questionnaire 

7. Student Questionnaire 

8. School Board Questionnaire 



Source Page 

Independent Study Progrann 113 
Hannden-New Haven, Connecticut 

Geornia, Governor's Honors Progrann 117 

Project Gifted, 121 
Warwick, Rhode Island 

Ocean View School District, 123 
Huntington Beach, California 

Illinois Gifted Progrann 125 

Illinois Gifted Progrann 129 

Illinois Gifted Progrann 133 

Illinois Gifted Progrann 135 
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SATIS-Q 

STUDENT ATTITUDE TOWARD INDEPENDENT 
STUDY QUESTIONNAIRE 

by 

Joseph S. Renzulli 
Robert K. Gable 

Bureau of Educational Research - 
University of Connecticut 



THIS IS NOT A TEST 
. . . there are no right or wrong answers 



Directions: 

On your answer sheet please record the following information in the blank 
spaces: 

1. Your school in the space marked "School'' 

2. Your grade ia the space marked "Grade" 

3. Number of years in the ISP in the space marked "Test" 

4. Subject areas you have studied in the space marked "Instructor" 

DO NOT WRITE YOUR NAME ON THE QUESTIONNAIRE 
OR THE ANSWER SHEET, NO ATTEMPT Wl LL BE MADE 
TO IDENTIFY PERSONS COMPLETING QUESTIONNAIRES, 

Following is a list of factors which are important in the effective operation 
of the Independent Study Program (ISP). Please rate the program on each of the 
factors by darkening the appropriate space on your separate answer sheet. Use 
what you would consider to be the ideal program as a standard of excellence in 
making your ratings. Keep in mind that there are no right or wrong answers to 
any of these questions. Your honest opinion about each >f actor will be the best 
answer. 

If the program is EXTREMELY POOR with respect to the factor 
darken space A on your answer sheet. 

If the program is BELOW AVERAGE with respect to the factor 
darken space B on your answer sheet. 

If the program is BELOW AVERAGE with respect to the factor 
darken space C on your answer sheet. 

If the program is ABOVE AVERAGE with respect to the factor 
darken space D on your answer sheet. 

If the program is EXCE LLENT with respect to the factor darken 
space E on your answer sheet. 



Prepared for the Independent Study Program for Gifted Students, Hamden-New Haven 
Cooperative Education Center, 1450 Whitney Ave., Hamden, Connecticut 06517. 
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A 
B 
C 
D 
E 



Above Average 
Excellent 



Extremely Poor 
Below Average 
Acceptable 



1. Extent to which the ISP has influenced you to attend college. 

2. Extent to which your ISP studies have influenced the area that you would 
like to major in if you go to college. 

3. Extent to which your ISP studies have influenced your career choice. 

4. Extent to which ISP has helped you develop skills in decision m||:ing. 

5. Extent to which ISP has helped you in making choices. 

6. Extent to which ISP has helped you to develop a sense of control over your 
future. 

7. Extent to which ISP has made you more excited about learning. 

8. Extent to which ISP has increased your motivation to learn. 

9. Extent to which ISP has helped you to become a better evaluator of your 
own work. 

10. Suitability of the method by which students are selected for the program. 

11. Opportunity to select topics for study which are of interest to you. 

12. Opportunity. to pursue topics to the extent that you desire. 

13. Degree to which the objectives of the program were in agreement with your 
personal objectives. 

14. Effect of the ISP on your study habits. 

15. Extent to which ISP has helped you to think critically. 

16. Extent to which ISP has helped you to organize your thoughts. 

17. Extent to which ISP has helped you to focus your thoughts. 

18. Extent to which the ISP has helped you to become acquainted with own 
thinking, working, and learning styles. 

19. Appropriateness of the way the ISP is organized. 

20. Extent to which ISP has helped you to develop more individual responsibility. 

21. Your overall rating of the ISP in terms of fulfilling your immediate educational 
needs. 

22. Extent to which you have mastered your objectives as a result of being in ISP. 

23. Degree to which you are challenged by ISP. 

24. Opportunity for interaction with your ISP teachers. 

25. Opportunities to meet with consultants or experts in the field(s) in which you 
are studying. 

26. Opportunities for you to express your ideas and feelings. 

27. Extent to which ISP has helped you to develop your self-confidence. 
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Answer questions 28 through 39 by: 



Darkening space A if you think the ISP is the best answer. 
Darkening space B if you think the regular school program is the 
best answer. 

Darkening space C if there is no difference between the ISP and 
your regular school program. 

28. Which holds you more responsible for work? 

29. In which do you try out ideas more? 

30. In which do you use your time to best advantage? 

31 . In which do you express more creativity? 

32. Which provides better teachers with highest ability? 

33. In which do teachers take the greatest personal interest in you? 

34. In which are teachers more enthusiastic about their subjects? 

35. In which do you put forth the greatest effort? 

36. Which challenges you the most? 

37. Which do you think is more effective in preparing you for courses that will 
follow the course(s) you are now taking in ISP? 

38. Which do you think is more effective in preparing you for acceptance into 
college? 

39. Which do you think is more effective in preparing you for the work that you 
will do in college? 
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Governor's Honors Program 
STAFF SEMANTIC DIFFERENTIAL 

Birthdate: 

Month Date Year 

Nunnber of years Teaching experience: 



Music 
Science 
Social Science 
Physical Education 
Counseling 

Other (specify) 

Date: 

The purpose of this activity is to measure the meanings of certain concepts 
related to the Governor's Honors Program by asking you to judge them against a 
series of descriptive scales. On each page you will find a different concept in 
parenthesis to be judged and beneath it a set of scales. You are to rate each concept 
on each of these scales. 

Here is how you are to use the scales: 

If you feel a particular concept is very much like one end of the scale, you 
should place your check mark as follows: 

PLEASANT X : : : : : : :UNPLEASANT 

1 2 3 4 5 6 7 

or 

PLEASANT : : : : ; ; X :UNPLEASANT 

1 2 3 4 5 6 7 

If you feel a particular concept is quite closely like one or the other end of 
the 5 :ale (but not extremely), you should place your check mark as follows: 

RUGGED : X : : : : : :DELICATE 

1 2 3 4 5 6 7 
or 

RUGGED : : : : : X : :DELICATE 

1 2 3 4 5 6 7 



From: Payne, D. A. Evaluation of the State of Georgia's Governor's Honors Program. Athens: 
University of Georgia, 1972. 



Check your teaching or specialization area. 

06. 

07. 

08. 

. 09. 

10. 

11. 



01. 


Art 


02. 


Drama 


03. 


English 


04. 


Foreign Language 


05. 


Mathematics 
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If you feel a particular concept is only slightly like one side as 
opposed to the other side (but is not really neutral), then you 
should check as follows: 

SHARP : :_>L • • • • -DULL 

1 2 3 4 5 6 7 

or 

SHARP : : : :_>L • • -DULL 

1 2 3 4 5 6 7 

If you consider the concept to be neutral on the sacle (both sides of 
the scale equally associated with the concept) or if the scale is com- 
pletely irrelevant (unrelated to the concept), then you should place 
your check mark in the middle space: 

HAPPY : : : X : : : :SAD 

1 2 3 4 5 6 7 



The direction toward which you check, of course, depends upon which 
. of the two ends of the scale best describes your feeling about each 
concept. 

Do not worry or puzzle over any one scale. It is your first impression, your 
immediate feeling about each concept that we want. On the other hand, please do 
not be careless, because we want your true impressions. Do not try to remember 
how you checked similar items earlier in the scale. MAKE EACH ITEM A 
SEPARATE AND INDEPENDENT JUDGMENT. 



Remember, you are judging the concept as you see it-not what we think or 
what others think. 



IMPORTANT: (1) Place your check marks in the middle of the spaces, 
not on the boundaries: 

this not this 
• -^L. ■_• • •__X : 

(2) BE SURE TO CHECK EVERY SCALE; DO NOT 
OMITANY. 

(3) NEVER PUT MORE THAN ONE CHECK MARK 
ON A SINGLE SCALE. 
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(INDEPENDENT ST' IDY) 



LARGE 

UNPLEASANT 

FAST 

DULL 

THIN 

HAPPY 

WEAK 

GOOD 

MOVING 

UNFAIR 

PASSIVE 

HEAVY 



:SMALL 



2 
2 
2 
2 
2 
2 
2 
2 
2 
2 



3 

T 



3 
3 
3 
3 
3 
3 
3 
3 
3 
3 



4 

T 



4 
4 
4 
4 
4 
4 
4 
4 
4 
4 



5 

"5" 



5 
5 
5 
5 
5 
5 
5 
5 
5 
5 



6 

T 



6 
6 
6 
6 
6 
6 
6 
6 
6 
6 



:PLEASANT 



:SLOW 



:SHARP 



:THICK 



:SAD 



:STRONG 



:BAD 



:STILL 



:FAIR 



:ACTIVE 



:LIGHT 



This scale was also used to measure the following concepts: Learning, Governor's Honors 
Program, Governor's Honors Program Seminars, Academically Talented Student, Artistically 
Talented Student, Dormitory Living, Audio-Visual Materials, Teachers, and Testbooks. 

Students filled out an idential set of scales. 
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PROJECT GIFTED-WARWICK 
1972-1973 



Parent Questionnaire 

Directions: Please do not sign your name to this questionnaire. No attempt will 
be made to identify persons completing these forms. Please return the question- 
naire in the enclosed envelope within the next two or three days. 
You can help to make Project Gifted a better program by giving careful thought 
to each of the questions that follow. Because of the relatively small number of 
persons involved in the project, each person's opinions will weigh heavily in 
analyzing the results. We appreciate your cooperation and assistance in helping 
us to evaluate Project Gifted. 

Yes No 

1. Have you been provided with enough information about 

why your child was selected for Project Gifted? 

2. Have you been provided with enough information about 

the objectives of Project Gifted? 

3. Have you been provided with enough information about 
the activities and experiences that your child pursues in 

Project Gifted? . 

4. Have you been invited to visit the Project Gifted clasv 

room? 

5. Have you been offered sufficient opportunity to 

discuss your child's progress with the teacher? 

6. Which of the following comments best expresses your 
child's general attitude about being in Project Gifted? 

Enthusiastic 

Positive • 

Indifferent 

Negative 

7. Has your child expressed pleasure or enjoyment about 
the work that he or she does in Project Gifted? (check 
one) 

Often 

Sometimes 

Seldom 

Never 

8. Which of the following statements best expresses your 
child's attitude toward the degree of challenge of the 
work in Project Gifted? 

Very challenging 

Somewhat challenging 

Not at all challenging 

No answer 



From: Renzulli, J. S. An Evaluation of Project Gifted. Storrs: The University of Connecticut, 
1973. 



Yes No 

9. Has your child encountered any problems with his 
friends as a result of being involved in Project Gifted? 
If yes, please describe. 

10. Do you think that the nanne "Project Gifted" calls 
unnecessary attention to the children participating in 

the progrann? 

1 1 . Have your or your child encountered any problenns as 
a result of being transported to another school? If yes, 

please describe. 

12. Has your child expressed a concern about nnissing work 
in the regular class or making-up assignments because 

he is out of the room to attend Project Gifted? 

13. Has your child's regular classroom teacher(s) expressed 
any displeasure because your child has missed work 
because he or she is attenting Project Gifted? If yes, 

please describe. 

14. If your child is invited to participate in Project Gifted 
next year, will you encourage him to do so? If no, tell 

why. 

15. Can you identify any changes in your child's behavior 
or attitude toward school or education which seem to 
result from his or her participation in Project Gifted? 

If yes, please describe. 

16. Do you have any specific suggestions for changes in the 
operation of Project Gifted or the way it affects 

children or their parents? 

Number of years your child has been in Project Gifted 
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PLEASE RETURN TO PUPIL PERSONNEL 
SERVICES BY FRIDA Y, APRIL 19 



OCEAN VIEW SCHOOL DISTRICT 
MENTALLY GIFTED MINOR (MGM) PROGRAM 



Check One (1) 

NEIGHBORHOOD SCHOOL 



ROBIN WOOD 



RANCHO VIEW 



Agree Neutral Disagree 
1 2 3 4 5 



1. I am (we are) satisfied with the overall 
MGM program now being offered my 
youngster. 

2. The school adequately keeps us 
informed regarding MGM activities my 
youngster(s) is/are involved. 

3. What form of communication would be most effective? Check One (1) 
(a) A Newsletter 

(b) Regularly scheduled parent-teacher conference 

(c) Individual, situational parent-teacher letter(s) 

(d) Situational telephone conferences 

(e) Scheduled, quarterly MGM parent meetings (evenings) 

4. In a questionnaire sent to M6I\/I parents last year, the following goal statements 
were selected in the following descending order by parents of MGM students. 
Would you please rank order the five (5) most important goal statements 
according to your perceptions. 

Please select the top f/Ve /5/ goahstatements as you see them relating to MGM 
students. Use 1 for your top priority, 2 for your second choice, etc. 

GOALS 

To develop a positive attitude ' 



a. Learning Skills: 



b. Self-image:! 



c. Self-realization: 



d. Reading: 



toward learning and a process for 
systematic problem solving. 

To build a positive, realistic self- 
image in order to accept the rewards 
of success and the consequences of 
failure. 

To recognize one's self-worth and 
develop an awareness of one's potential 
contribution to society. 

To develop the skills of reading to the 
best of one's ability in order to enjoy 
the benefits of reading. 



From: Ocean View School District, Huntington Beach, California. 
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e. Ecology: 

f. Citizenship: 

g. Physical Health: 

h. Communication Skills: 

i. Moral and Spiritual Values: 
j. Mathematics Skills: 

k. Social Values: 

s 

I. Vocational Competency: 

m. Science Skills: 

n. Fine Arts: 

o. Social Studies: 

p. Use of Leisure Time: 



GOALS 

To respect, conserve, and enjoy 
nature's contributions to our ways 
of life. 

To be a responsible, participating 
member of society. 

To develop and maintain a healthy body. 

To listen carefully and express oneself 
effectively through written and oral language. 

To develop a sense of honesty and fairness 
wih respect toward other people. 

To develop competency in math skills to the 
best of one's ability and apply them to every- 
day life. 

To adjust easily to social relationships and be 
tolerant of the opinons and actions of others. 

To value the personal and material rewards 
gained from a well chosen vocation. 

To appreciate the value of scientific discoveries 
and apply scientific skills to everyday life. 

To express, develop, and appreciate artistic 
talent (painting, music, dance, sculpture, etc.) 

To understand the consequences of history on 
the past and present peoples of the world. 

To develop the skills necessary to enjoy a 
balance between relaxation and productive 
leisure time activities. 



In a questionnaire sent home to MGM parents last year, the following represent . 
the ten (10) most frequently cited MGM highlights that MGM youngsters shared 
unsolicited with their parents. Would you choose the apparent favorite MGM 
activity your child shared with you this year? 



a; A particular field trip 

b. A science activity 

c. An oceanography activity 

d. An outside speaker 

e. A play 



f. A study of countries 

g. An arts and crafts project 

h. His/her reading program 

i. A creative writing project 
j. Photography/film making 



k. (other) " 

For those of you who are inclined, please add any constructive comments that will 
assist the district to meet your expectations regarding MGM and your child(ren). 
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INSTRUMENT FOR TEACHERS 



We suggest you read through the questionnaire before answering any items. 
Part I 

Please indicate the extent to which you agree or disagree.with each of the following 
statements by circling the appropriate letter. The letters mean the following: 

SA — Strongly agree 

A — Generally agree 

U — Undecided or neutral 

D — Generally disagree 

SD — Strongly disagree 

Please use the comment line if you want to explain your answer. 

1 . I feel this school does a good job with the SA A U D SD 

able students. 4 
Comment 



2. The teaching situation in this school is a SA A U 

favorable one. 3 1 

Comment 



SD 



3. This school provides a well-balanced SA A U 

educational program. 3 1 

Comment 



SD 



4. The community is very supportive of our SA A U 
school program. 1 2 1 

Comment 



SD 



5. Students in the gifted classes spend too much SA A 
much time on those classes at the expense 
of other classes. 

Comment 



U 



D SD 
4 



6. Many students not in the gifted program SA A U D SD 

resent the program. 1 111 

Comment 



From: Sjogren, D., Hopkins, T., & Cooler, D. Evaluation Plans and Instruments: Illustrative 
Cases of Gifted Program Evaluation Techniques. Center for Instructional Research 
and Curriculum Evaluation. Urbana-Champaign: University of Illinois. 



7. I think more students should be in the SA A U D SD 
program than at present. 1 2 1 

Comment 



8. Generally the students in the gifted SA A U D SD 

program are the ones who should be in it. 1 2 1 
Comment 



9. I feel lam well-trained for teaching a SA A U D SD 

gifted class. 3 1 

Comment 



10. I feel the gifted program is meeting its SA A U D SD 
objectives quite well for the students 1 2 1 

who are in it. 
Comment 



11. lam concerned that I spend too much SA A U D SD 
time preparing for and working with the 1 3 

gifted. 
Comment 



12. I have adequate materials and equipment SA A U D SD 

to teach as I want to teach. 2 2 

Comment 
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Part II 



1 . Suppose this school were to have a windfall of money. How would you 
recommend the money be Spent? Please indicate by ranking the following 
programs in terms of which have need for more materials, staff, etc. The 
program may or may not exist in your school now. iSive a rank of one to 
the program that you consider most important, a rank of two to the next 
most important, and so on until you have ranked all. 

4 Vocational or pre-vocational program (shop, home ec, etc.) 

1 Academic programs 

8 Extra-curricular activities 

2 Gifted program 

7 Pupil personnel services (includes guidance, health services, etc.) 

5 Program for low-achieving students 

6 Administrative support (includes clerical support) 

3 Library 

Other (please specify) Resource Center 



2. Now let's think the other way. Suppose the budget of your school were to 
be cut. How would you recommend the cuts be made? Please indicate by 
ranking the following programs in terms of their expendability from your 
point of view. Give a rank of one to the program that you feet should be 
cut or reduced first, a rank of two to the next that could be cut or reduced, 
and $0 on until you have ranked all. 



6 


Library 


6 


Academic programs 


1 


Extra-curricular activities 


4 


Gifted program 


2 


Pupil personnel services 


6 


Lunch program 


3 


Administrative support 




Other (please specify) 
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Part III 

1. Write a brief description of the eighth grade classes that you teach this 
year. Indicate the topics covered and the kinds of activities that have been 
carried out. 



2. Indicate the ways your teaching of the gifted class differs from the other 
class with respect to the following: 

a. Objectives. 



b. Topics studied 



c. Techniques used 



3. From your experience with classes of gifted students, what skills and attitudes 
seem to be most important for an instructor to have? 



4. Describe any teaching situations in your lifted class this year where you felt 
especially pleased with the results. 



5. What anecdotal evidence do you have of the impact of the gifted program? 
(Observed changes in students, testimonials by students, parents, etc.) 
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PARENT QUESTIONNAIRE 



We suggest you read through the questionnaire before you start to answer. Respond 
in terms of your oldest or only child now in School D. 
I. Who completed the questionniare? (check one) 
Father (or male guardian) only 

Mother (or female guardian) only 

Both parents 

Other 



II. Please indicate the extent to which you agree or disagree with each of 
the following statements by circling the appropriate letter. The letters 
mean the following: 

SA — Strongly agree 

A ~- Generally agree 

U — Undecided or neutral 

D — Generally disagree 
SD — Strongly disagree 

Please use the coment line if you want to explain your answer. 

1 . School D provides a well-balanced SA A U D SD 

educational program. 2 9 

Comment 



2. This school has a good program for SA A U D SD 
the able students. 3 7 1 

Comment 



3. Most of the teachers our child has SA A U D SD 

had at School D have done a good 1 10 

job. 
Comment 



4. ^ Most parents feel School D is a SA A U D SD 

good school. 18 2 

Comment 



5. , Our child seems to like this school. SA A U D SD 

(As well as any child likes school.) 4 7 
Comment 



ERIC 



From: Sjogren, D., Hopkins, T., & Cooler, D. Evaluation Plans and Instruments: Illustrative 
Cases of Gifted Program Evaluation Techniques. Center for Instructional Research 
and Curriculum Evaluation. Urbana-Champaign: University of Illinois. 
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6. Students at School D are receiving SA A U D SD 

a good education in the basics like 

math, English, history, and science. 2 9 
Comment 



7. There are too many frills in the SA A U D SD 

School D program. 1 2 8 

Comment 



8. School D has a good extra-curricu- SA A U D SD 
lar program, i.e., athletics, nj^gsic, 1 9 1 
school paper, drama, clubs, etc. 

Comments 



9. School D should have more SA A U D SD 

vocational-type courses. 3 2 4 2 

Comment 



III What do you regard as the strengths (the best things) of School D? 



IV. What suggestions do you have for further strengthening the school? 
(Things that might be added, increased, dropped, etc.) 



V. Please answer each of the following questions about classes your 
child is now taking in the following subject areas: math, science, 
social studies, language arts. 

la. Which class does your child like the best? 

lb. Which class do^ your child like the least? 

2a. In which class is your child learning the 
most? 

2b. In which class is your child learning the 
least? 

3a. On which class does your child work the 
hardest? 

3b. On which class does your child work the 
least? 
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VI. Is your child in a special program in the school? ( ) yes 

( ) no 

VII. (If yes to VI) What is the program and what is your opinon of it? 



VII I. (If no to VI) What special program would you like to see provided 
that your child could be in? 
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STUDENT QUESTIONNAIRE 



Part 



Check the extent to which you agree or disagree with the following statennerts 
by circling the appropriate letter. The letters mean the following: 

SA — Strongly agree 

A — Generally agree 

U — Undecided or neutral 

D — Generally disagree 

SD — Strongly disagree 

Use the comnnent line if you want to explain your answer. 

1 . Most days I enjoy school 
Connnnent 



2. I like the way our classes are run 

in separate sections. 
Comment 



3. I am learning a lot in school this 

year. 
Comment 



4. Most of the kids in this school are 

friendly with each other. 
Comment 



5. I think school is hard this year. 
Comment 



6. My parents are satisfied with my 

school work this year. 
Comment 



SA 


A 




n 

u 


SD 




6 


4 


0 


0 


00 


Gifted 


2 


10 


1 


1 


1 


Regular 


SA 


A 


11 


n 

u 


SD 




7 


2 


1 


1 

1 


0 


a if tpH 


4 


5 


3 


1 


2 


Regular 


SA 


A 


u 


D 


SD 




4 


7 


0 


0 


0 


Gifted 


3 


9 


3 


0 


0 


Regular 


SA 


A 


U 


D 


SD 




6 


3 


2 


0 


0 


Gifte.d 


8 


4 


1 


1 


11 


Regular 


SA 


A 


U 


D 


SD 




0 


2 


.':-3;; 


5 


1 


Gifted 


1 


2 


3 


5 


4 


Regular 


SA 


A 


U 


D 


SD 




2 


6 


2 


1 


0 


Gifted 


3 


6 


3 


1 


2 


Regular 



7. 1 don't have time to participate in 


SA 


A 


U 


D 


SD 


as many things as 1 would like. 


1 


2 


1 


5 


2 


Comment 


3 


2 


3 


2 


5 



Gifted 
Regular 



From: Sjogren, D., Hopkins, T., Cooler, D. Evaluation Plans and Instruments: Illustrative 
Cases of Gifted Progran) Evaluation Techniques: Center for Instructional Research 
and Curriculunn Evaluation. Urbana-Channpajgn: University of Illinois. 
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9. 1 wish 1 could be in some of the 


SA 


A 


U 


D 


SD 




other classes or sections. 


1 


1 


3 


4 


2 


Gifted 


Comment 


3 


4 


1 


4 


3 


Regular 


1 0. A lot of the stuff we study in 


SA 


A 


U 


D 


SD 




school isn't very important. 


0 


0 


3 


4 


4 


Gifted 


Comment 


1 


2 


3 


5 


4 


Regular 



Part III 

1 . In what ways has your school work been different this year from other years? 



2. What three things do you like best about school? 



3. What three things do you like least about school? 



4. What would you like to have in this school that you don't have now? 
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School E 
Form Four 

SCHOOL BOARD QUESTIONNAIRE 

We suggest you read the questionnaire before you start to answer. 

I. Please indicate the extent to which you agree or disagree with each of 
the following statements by circling the appropriate letter. The letters 
mean the following: 

SA — Strongly agree 

A — Generally agree 

U — Undecided 

D — Generally disagree 

SD — Strongly disagree 

Please use the comment line if you want to explain your answer. Answer 
questions 9 and 10 only if you have children in the high school. 



1. 


School E High School provides a well- 
balanced educational program. 
Comment . 


SA 


A 


U 


D 


SD 


2. 


This school has a good program for the 

able students. 

Comment 


SA 


A 


U 


D 


SD 


3. 


Most parents feel School E is a good 

school. 

Comment 


SA 


A 


U 


D 


SD 


4. 


The School E program is mainly for the 

college bound student. 

Comment 


SA 


A 


U 


D 


SD 


5. 


Students at School E are receiving a 
good education in the basics like math, 
English, history, science. 
Comment 


SA 


A 


U 


D 


SD 


6. 


There are too many frills in the School 

E program. 

Comment 


SA 


A 


U 


D 


SD 


7. 


School E has a good extra-curricular 
prograr% i.e., athletics, music, school 
paper-, drama, clubs, etc. 
Comment 


SA 


A 


U 


D 


SD 


8. 


School E should have more vocational 

courses. 

Comment 


SA 


A 


U 


D 


SD 


9. 


Most of the teachers our child has had 
at School E have done a good job. 
Comment 


SA 


A 


U 


D 


SD 



From: Sjogren, D., Hopkins, T., & Cooler, D. Evaluation Plans and Instruments: Illustrative 
Cases of G if ted Program Evaluation Techniques. Center for Instructional Research 
and Curriculum Evaluation, Urbana-Champaign: University of Illinois. 



10. Our child seems to like this school. SA A U D SD 
(as well as any teenager likes school.) 

Comment^ 

II. What do you regard as the strengths (the best things) of School E? 



III. What suggestions do you have for further strengthing the school? (Things 
that might be added, increased, dropped, etc.) 



IV. Suppose this school were to have a windfall of money. How would you 
recommend the money be spent? Please indicate by ranking the follow- 
ing programs in terms of which have need for more materials, staff, etc. 
The programs may or may not exist in your school now. Give a rank 
of one to the program that you consider most important, a rank of two 
to the next most important, and so on until you have ranked all. 
Rank 

Vocational program 

Academic programs 

Extra-curricular activities 

Gifted program 

Pupil personnel services (includes guidance, health services, etc.) 

^ — Program for low-achieving students 

Administrative support (includes clerical support) 

Other (please specify) 

V. Now let's think the other way. Suppose the budget of your school were 
to be cut. How would you recommend the cuts be made? Please indicate 
by ranking the following programs in terms of their expendability from 
your point of view. Give a rank of one to the program that you feel should 
be cut or reduced first, a rank of two to the next that could be cut or 
reduced, and so on until you have ranked all. 
Rank 

Vocational program 

Academic programs 

Extra-curricular activities 

Gifted program 

Pupil personnel services 

Lunch program 

Administrative support 

Other (please specify) I _Z 
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APPENDIX B 



Sample Non-Test Instruments for Assessing 
Student Performance 





Title 


Source 


Page 


1. 


Evaluation Scale for Visual 
Arts 


Project Gifted, 
Warwick, Rhode Island 


139 


2. 


Evaluation Scale for Creative 
Writing 


Project Gifted, 
Warwick, Rhode Island 


141 


3. 


Parent Evaluation of Pupil 


California Gifted Program 


143 


4. 


Teacher Evaluation of Pupil 


California Gifted Program 


145 


5. 


Pupil Self-Evaluation 


California Gifted Program 


147 


6. 


Student Evaluation 


Learning Center Program, 
Hillsborough County, Florida 


149 


-J 
1 . 


Student Evaluation Scale 


^/l^k/Hpl IntPrmpHifltp Fnrirhmpnt 

iVi VJLiC i III LC 1 1 1 ICvJ la b. 1 1 1 IV«I 1 1 1 Id 1 L 

Program, Great Falls, Montana 


151 


8, 


Self-Esteem Inventory 


Great Falls Public Schools, 
oreai raiis, iviontana 


157 


9. 


Parent Evaluation of Pupil 
(Elementary) 


State Program for Mentally 
Gifted Minors, 
Coronado, California 


159 


10. 


Student Evaluation Scale 
(Elementary) 


State. Program for Mentally 
Gifted Minors, 
Coronado, California 


161 


11. 


Teacher's Evaluation Form 


State Program for Mentally 
Gifted Minors, 
Coronado, California 


163 
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WARWICK PUBLIC SCHOOLS 
WARWICK, RHODE ISLAND 

PROJECT GIFTED 
Evaluation Scale for Visual Arts 

■* 

Code No. Age Boy/Girl TOTAL SCORE 

Evaluator Position Date 



Elements 


Low 


Moderate 


High 


1 


2 


3 


4 


5 


6 


1. Creative Expression, Imagination, 
Uniqueness 














II Flexihilitv Aooreciation and Adaota- 
bility to Various Media 














III. Fluency Variety of Number of Ideas 














IV. Sensitivity-Composition-Design 














V. Manipulative Skills Construction- 
Weaving, etc. 














VI. Growth 














Column Total 
Weight 
Weighted Column Total 














1 


2 


3 


4 


5 


6 
















TOTAL SCORE 





From: Renzulli, J.S. An Evaluation of Project Gifted. Storrs. University of Connecticut, 1973. 
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WARWICK PUBLIC SCHOOLS 
WARWICK. RHODE ISLAND 

PROJECT GIFTED 
Evaluation Scale for Creative Writing 

TOTAL SCORE 
Date 





Low 


Moderate 


High 




1 


2 


3 


4 


5 


6 


1. Organization 

1 lnit\/ Ho\/olonrnont pl^ritv/ 














II. Mechanics 

Structure, grannnner, usage 
Skills: spelling, punctuation 














\\\\ Originality 

Creative f luence, emotional quality, 
Innagination, style, theme 














IV. Unusual/Other Elements 

Dialogue, special format mood/ 
setting, character development 














V. Growth 














Column Total 














Weight 


1 


2 


3 


4 


5 


6 


Weighted Column Total 
















TOTAL SCORE 





Code No. Age Boy/Girl 

Evaluator Position 



From; Renzulli, J, S.An Evaluation of Project Gifted. Storrs, University of Connecticut, 1973. 
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PARENT EVALUATION OF PUPIL 



Dear Parents: 

We need to impose upon you for a final time to get your reaction to your 
child's experiences as a participant in the State Study. It is highly important to 
us to have your evaluation of the program's effectivenes.. 

We shall appreciate it very much if you will fill out the attached page and 
return it as soon as possible to 



Thank you very much. 

PARENT EVALUATION OF PUPI L 

Pupil's Name 

. Will you please think of your child at the present time in comparison to 
last year. As a result of his participation in the State Study, please rate him on 
the following items. Place the letters a, b, c, d, and e on the line following each 
item according to the scale below. You may have difficulty in responding to 
some of the items. Please make the best estimate that you can. 

(a) Much less (b) Less (c) About the same (d) More (e) Much more 

1. Ability to think things through'for himself 

2. Knowledge of subject matter areas (science, social studies, and 

others he has taken) 

3. Interest in school 

4. Ability to see relationships 

5. Ability to find information — 

6. Ability to work well by himself 

7. The liking and respect of other pupils for him 

8. Ability to judge the usefulness of facts 

9. Ability to get along well with his teacher(s) 

10. Enjoyment of learning 

1 1 . Knowledge of arithmetic, spelling, and other basic skills .... 

1 2. Curiosity about learning new things 

13. Ability to accept responsibility 

14. Opportunity to make things, experiment, and use ideas .... 

15- Knowledge of his strengths and weaknesses 

16. Willingness to do work as a leader 



From:. Simpson, R. E., & Martinson, R. f^. Educational Program for Gifted Pupils. 
Sacramento, California: California State Department of Education, 1961. 
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Please answer the questions that follow 

17. Has participation in the study helped him or her? (Yes or no) 
Please explain. 



18. Has participation created problems for hi:n or her? (Yes or no) 
Please explain. 



19. Would you like to have the program continued? (Yes or no) 
Please explain. 



20. What changes, if any, would you suggest? 



Name 

Address 

Telephone* 
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TEACHER EVALUATION OF PUPIL 



Pupil Teacher Date 

Please evaluate this pupil by placing the letter a, b, c, d, e on the line 
following each itenn according to the scale below. Think of hinn in relation 
to his perfornnance at the start of the progrann. 

(a) Marked loss . (d) Increasing 

(b) Dinninishing (e) Marked increase 

(c) About the sanne 

1 . Ability to solve problenns — 

2. Knowledge of subject nnalter areas — 

3. Interest in school . • • • — 

4. Ability to think in ternns of the whole and to see parts in relation 

to the whole — 

5. Research skills — 

6. Ability to work independently — 

7. Status in peer group . — 

8. Critical thinking ability — 

9. Rapport with teacher — 

10. Motivation toward learning — 

1 1 . Knowledge of basic skills (fundamentals) — 

12. Intellectual curiosity — 

13. Ability to accept responsibility — 

14. Opportunity to create and experiment with ideas and things . . — 

15. Self-understanding — 

16. Acceptance of leadership roles — 



From : Simpson, R. E., & Martinson, R. A. Educational Program for Gifted Pupils. 

Sacramento, California: California State Department of Education, 1961. 
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PUPIL SELF-DEVALUATION 



Pupil's Name 

Will you please think of yourself at the present time in comparison to 
last year. As a result of this year's work, please rate yourself on the following 
items. Place the letters'a, b, c, d, and e on the line following each item 
according to the scale below. 

(a) Much less (b) Less (c) About the same (d) More (e) Much more 

1 . Ability to think things through for yourself 

2. Knowledge of subject matter areas (science, social studies, and others 

I have taken) 

3. Interest in school 

4. Ability to see how things go together in a situation (see relationships) 

5. Ability to find information 

6. Ability to work well by myself 

7. The liking and respect of other pupils for me 

8. Ability to judge the usefulness of facts 

9. Ability to get along with my teacher(s) 

10. Enjoyment of learning . 

11. Knowledge of arithmetic, spelling and other basic skills 

1 2. Curiosity about learning new things 

13. Ability to accept responsibility 

14. Opportunity to make things, experiment, and use ideas 

1 5. Knowledge of my strengths and weaknesses 

16. Willingness to do work as a leader ^ 

Please answer the questions that follow: 

17. Has the school year been helpful to you? Yes No 

Please explain. 



18. Has any part of the school work this year created any problems 

for you? Yes No. 

Please explain. 



From: Simpson, R. E., & Martinson, R. A. Educational Program for Gifted Pupiis. Sacramento, 
California: California State Department of Education, 1961. 



142 



19. Would you like to continue in a group like the one you had 

this year? Yes No, 



20. What changes, if any would you suggest? 



148 143 

ERIC 



STUDENT EVALUATION 
LEARNING CENTER PROGRAM 



For the past sixteen weeks you have been attending sessions at the Hillsborough 
County Learning Center. We would like to know some of your feelings about the 
program. By answering questions and completing the following sentences, you can 
help us in improving the program. 

1. Which class did you like best? 

2. Why? 

3. Of the classes I was not in, I wish I could have taken 

4. Why? 

5. I wish my classes at the Learning Center were longer, shorter, , 

the same_ (check one) 

6. The Learning Center needs more 

7. The class in which I learned or accomplished most was ■ 

8. If I could change three things about the Learning Center, I would 

a. 

b. 

c. 

9. Has the Learning Center helped you in any way with things you do at 
sc hoo I ? 

10. How? 

1 1 . Has the Learning Center helped you in any way with things you do at 
home? 

12. How? 

13. Has the Learning Center helped in any way with the way you get along 
with or feel about people? . If so, how? 



From : Florida's State Resource Mar)ual for Gifted Child Education. State of Florida Department 
of Education, 1973. 
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-INSTRUCTIONS FOR USE OF STUDENT EVALUATION SCALE- 



Model Intermediate Enrichment Center 
INTRODUCTION 

The s.udent evaluaiion scale is for rating student personality traits which may 
be associated with academic success. This scale was originally designed by August 
Dvorak, Carroll Mjelde, and Alden Nickelson at the University of Washington. It 
was field tested in the public schools and rigorously designed for high reliability/ 
provided the rater takes time to read the directions and follows them. 

PURPOSE 

The major purpose of this scale is to experimentally evaluate the non-intellectual 
traits which may be associated with academic success in the primary grades. This 
sacle is adapted for statistical treatment of its data. 

DIRECTIONS FOR USE OF THE SCALE 

1 . Take a few minutes before beginning the rating procedure and familiarize 
yourself with the "traits" and their definitions. 

2. Note that space is provided for indicating that particular "tracts" have not 
been observed for a given student. 

3. It is suggested that the rater evaluate all students for a particular "trait" 
before going on to the next trait. 

4. Note that there is a "high" and "low" rating for each set of cues on the 
scale. 

5. Fill in the blanks at the top of each rating scale. 

SOME COMMON RATING ERRORS 

Error of Central Tendency — Many teachers hesitate to give either extremely 
high or extremely low ratings. They tend to group their ratings close to the 
center of the scale. This error occurs most commonly with inexperienced raters. 

Logical Error — There is a tendency for evaluators to give similar ratings to 
qualities which seem to be related logically such as speaking ability and writing 
ability. The scale you will be using has been designed to reduce Logical Error 
to a minimum by selecting and defining the "traits" as discrete qualities. 

Halo Effect — Evaluators are sometimes unable to prevent their rating of one 
quality or "trait" from being influenced by other knowledge about the 
individual. As a result, the rating is shifted in the direction of a general impression. 
Halo may affect the rating of one quality, a number of qualities or "traits", or 
the over-all rating. 

Error of Personal Bids — This error, common to all human beings, enters into 
their evaluation of others. It is a human propensity to perceive and evaluate 
others on the basis of personal criteria and reference points. The Error of 
Personal Bias may be favorable or unfavorable to the individual being rated 
depending largely on the bias of the rater. 



From: Great Falls Public Schools, Great Falls, Montana. 



Error of Standards - Some evaluators tend to overrate or underrate everyone as 
compared with the average of other evaluators. They do this because of differences 
in their marking standards. Since each evaluator can use only his own standards, 
there is the possibility that there will be as many different standards as evaluators. 
However, through the use of a uniform rating scale in which the "traits" are clearly 
defined and cues are discrete and concise, the effects of Error of Standards can be 
eliminated or greatly reduced. 

DEFINITIONS 

The following are elaborations of what the authors conceived, and teachers are asked 
to accept, to be the various traits: 

Trait No. 1 - ALERTNESS -~ The mental wakefulness and organization which 
makes individuals sensitive to the significance of what they feel, smell, taste, hear, 
and see. It is opposite of being unconscious or semiconcious of what goes on, 
or is to go on, about them. 

Trait No. 2 - AMBITION - The eager, determined, all-pervasive desire for 
personal advancement, acquisition of knowledge, honors, superior achievement, 
and eventual accomplishment of formulated, high, purposeful goals demonstrated 
by planning and industrious effort. Ambition is not effortless daydreaming, but 
rather is a controlling drive pursued with eager, purposeful determination in and 
outside the classroom. 

Trait No. 3 - APPEARANCE AND BEARING - The individual characteristics 
which create positive or negative impressions about individuals as they walk, stand, 
or say "Good Morning." This trait involves only: (1) the students' cleanliness, 
finger nails, hair cutting, combing, (2) his dress (appropriateness, cleanliness, 
pressing, manner of wearing, shoe condition, etc.). A painter whose coveralls 
are paint spotted and patched, but clean and ironed or pressed is well dressed, a 
girl with a sable stole dragging or imappropriately worn is not, (3) posture and 
physjcal pdise, (4) address-frank, direct, neither hang-dogged nor smart alecky. 

Trait No. 4 - CREATIVITY-ORIGINALITY ^ Originality is primarily 
the mental capacity to be intelligently and constructively different, to search 
imaginatively for Galbreth's "One Best Way." Originality is neither perfect 
blind followership nor unorthodoxy for the sake. of being different. Creativity 
is essentially the doing, the building, the bringing into being, the results of 
originality. Desire, planning, execution and pleasure in accomplishment should 
be involved. Creativity and originality produce new books, cakes, chemicals, 
works of art, bridges, buildings, and beautiful interior decorations. 

Trait No. 5 - CURIOSITY AND INTEREST - The aggressive, inquisitiveness 
and anxiousness to learn through careful attention the "what, why, when, where, 
and how" of phenomena in and outside the classroom; the intellectual engrossment 
in exploration and investigation characterized by active, thorough, purposeful study; 
a lively involvement in satisfying a thirst for knowledge. 

Trait No. 6 - INDUSTRY - The ability and willingness to exercise dilligence, 
aggressiveness, steadfastness, persistence, and perseverence in application to 
assigned tasks; the degree of stick-to-itiveness exhibited toward the completion 
of a task; the amount, intensity,, and steadiness of effort expended toward a goal. 
Trait No. 7 - JUDGMENT - The ability to discriminate from data and 
effective reasoning the important elements and values of a situation and make 
sound decisions or conclusions; the ability to reach logical, accurate, and practical 
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decisions or conclusions by assembling, evaluating, comparing, all available 
data; the ability to anticipate, and discriminate between, alternate courses 
of thought and/or action. 

Trait No. 8 - SELF-CONFIDENCE - Self-reliance, self-assurance, 
certainty in own abilities, opinions, actions, and objectives in and outside the 
classroom; the freedom from unjustified fears and inhibitions due to realistic 
evaluation of own abilities, skills and strengths required for particular tasks, 
actions, or assignments. 

Trait No. 9 - DEPENDABILITY - The trait which enables those who 
work with, or depend on, individuals to know that their promises, obligations, 
and responsibilities will be fulfilled. 

Trait No. 10 - LEADERSHIP - The initiative and ability to inspire, direct, 
control, stimulate, and influence others toward group action; the ability to 
organize, manage, or direct the efforts of others effectively; skill and resource- 
fulness in organizing, initiating, and securing cooperation of others for group 
action in and outside the classroom. 

THE SCALE AND ITS USE 

Measuring devices for length, weight, electricity, liquids, etc., by law, have feet, 
yards, pounds, and gallons with exact definitions and specifications. A scale for 
measuring qualities or personal traits, to be usable, must have definitions and 
specifications. In rating scales, such as the "Student Evaluation Scalel-B", the 
definitions and specifications should create images comparable to the progressively 
better specimens in handwriting scales, or the color scales used by paint mixers 
or blenders. 



A.D., CM., A.N.-UofW 
MPihe 
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-MODEL INTERMEDIATE ENRICHMENT CENTER- 
STUDENT EVALUATION SCALE 1-B 



1 

ALERTNESS 

Attentiveness, vigliance, wake- 
fulness, sensitivenessjn and out- 
side the classroom; perceives and 
reacts to what he tees and hears. 

Not Observed X 


— Low— 

1 2 3 

Frequently inattentive, dis- 
tracted, preoccupied, and un- 
responsive; passive. Engages in 
program to limited degree. 
Absent-minded, frequently day- 
dreams; dull. Asks thoughtless 
questions. 


—Average— 

4 5 6 

Moderate attentiveness; active 
in the most interesting phases. 
Occasionally attention winders. 
Occasionally asks ques' ons, 
usually responds to questions. 


—High— 

7 8 9 

Completely absorbed and 
engrossed in work; gives his un- 
divided attention. Frequently 
asks pertinent intelligent ques- 
tions. Lively, responsive wide- 
awake; "catches-on" quickly; 
perceptive. 


2 

AMBITION 

Eagerness for knowledge, honors 
advancement, superior attain- 
ment; determined to achieve 
purposeful, high realistic goals. 

Not Observed X 


-High- 

9 8 7 

Resolute, decisive, definite, 
eager concerning personal 
accomplishment. Willing to 
sacrifice to achieve purposeful 
realistic, high goals. 


—Average— 

6 5 4 

Moderate determination for 
honors, knowledge, advance- 
ment, personal achievements. 
Average goals through moderate 
sacrifice. 


— Low— 

3 2 1 

Little desire for knowledge, 
personal improvement, content, 
to "get by," complacent "as is". 
Goals vague, unrealistic, indefin- 
ite or nonexistent. 


3 

APPEARANCE AND 
BEARING 

Personal grooming, cleanliness; 
neatness, appropriateness, and 
condition of clothing; posture, 
physical demeanor; composure; 
poise; mannerisms which create 
levels of impression. 

Not Observed X 


— Low— 
1 2 3 

Slovenly; neglects grooming and 
clothes; frequently dresses inap- 
propriately. Unconcerned about 
poor carriage, slouchiness, lack 
of coordination, awkwardness. 
Makes unfavorable impression. 
Room for much Improvement. 


—Average— 
4 5 6 

Usually exhibits orderliness, tidi- 
ness in dress and grooming; 
room for improvement. Blends 
into the group. 


-High- 
7 8 9 

Habitually careful about groom- 
ing, cleanliness, posture, 
condition and appropriateness 
of dress. Stands, sits, walks 
erect assumes an attentive posture. 
Stands out; creates highly favor- 
able impression by appearance. 


4 

CREATIVITY- 
ORIGINALITY 

Capacity and willingness for 
constructive, rational, unortho- 
dox, independent solutions and 
actions. Intelligent, inventive. 
Ingenious, productive, imagina- 
tive, or novel in thought and 
action. 

Not Observed X 


-High- 
9 8 7 

Ingenious, inventive, productive. 
Quick to devise, develop and 
test new approaches to problems 
Vivid, lively, fertile, practical 
imagination; formulates and 
tests worthwhile, original, novel 
ideas. 


—Average— 
6 5 4 

Usually accepts the orthodox; 
occasionally attempts original, 
constructive. Ingenious, thought 
and action. Ability to perceive 
areas for improvement varies; 
usually dependent upon direc- 
tions. 


— Low— 
3 2 1 

Unable to depart from the tradi- 
tional, or to generate new ideas, 
solutions, developments. Unimag- 
inative, uninventive. Definite 
dearth of new ideas. 


5 

CURIOSITY AND 
INTEREST 

Aggressive Inquisitiveness and 
desire to learn the "what, why, 
when, where, how" by 
thorough, active, and purposeful 
inquiry, study, and careful 
exploration and investigation. 
Not Observed X 


— Low— 
1 2 3 

Unreflective, uninquiring, 
narrow, indifferent. Disinteres- 
ted in schoolwork; uncorfcerned 
with intellectual pursuits; little 
or no interests in the "what, 
why, when, where, how." 


—Average— 
4 5 6 

Moderately interested in school 
subjects and world phenomena; 
asks questions. Inquisitiveness 
occasionally aroused in mani- 
pulative or exploratory activities 


-High- 
7 8 9 

Thoughtful, analytical curious, 
intellectually energetic, broad 
interests. Unfamiliar excites his 
inquisitiveness; asks penetrating 
questions; inspired to independent 
investigation. Definite thirst for 
knowledge. 
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STUDENT EVALUATION SCALE 1-B (Continued) 
Model Intermediate Enrichment Center 



6 

INDUSTRY 

Perseverance, aggressiveness, 
diligence, persistence, steadfast- 
ness in application to assigned 
tasks. Capacity and willingness 
to pursue tasks until completed. 

Not Observed X 


-High- 

9 8 7 

Outstanding aggressiveness, 
drive, perseverance, diligence, 
effort, persistence and 
application on purposeful 
tasks. Conscientious, tireless, 
energetic, enterprising, pains- 
taking worker. 


—Average— 

6 5 4 

Average diligence, steadfastness, 
perseverance, application to 
assigned tasks. Usually pursues 
tasks to completion; occasion- 
ally requires prodding, or 
stimulation. 


— Low— 

3 2 1 

Definite lack of effort, persistence, 
perseverance application, stead- 
fastness. Easily distracted, dis- 
couraged, requires continual 
prodding; Indifferent, wastes time, 
accomplishes little. 


7 

JUDGMENT 

Ability to reason from known 
date to logical decisions, con- 
clusions, to anticipate and 
evaluate results of alternative 
actions; to think ahead. 

Not Observed X 


— Low— 

1 2 3 

Uses careless, indiscriminate, 
thoughtless approaches to 
solving problems; neglects or 
misinterprets facts. Formulates 
fanciful, illogical, unsound, 
impractical, inexact,decisio*ns 
or conclusions. 


—Average— 

4 5 6 

Considers most facts, variables; 
usually sees relationships. 
Decisions usually based on 
logical reasoning; occasionally 
jumps to conclusions. 


-High- 

7 8 9 

Uses careful, discriminating, 
systematic, analytical approaches 
in making decisions; foresight 
and clarity in thinking. Consist- 
ently logical, practical, accepts 
decisions. 


8 

SELF-CONFIDENCE 

Seif-(sufficlency, dependence, 
reliance, assurance, sureness,) 
confident about own abilities, 
opinions, actions, and objectives. 

Not Observed X 


-High- 

9 8 7 

Balance self-( reliance, sureness, 
assurance, evaluation) confi- 
dence in own abilities, actions, 
opinions. Calm, poised, self- 
reliant, free from unfounded 
fears. 


—Average— 

6 5 4 

Usually reasonable self-(reliant, 
confident, assured). Occasionally 
disconcerted, ill-at-ease, or 
needs encouragement. 


— Low— 

3 2 1 

Overestimates own abilities, 
overconfident, conceited; lack- 
ing in modesty, humility; OR 
unduly hesitant, uncertain, timid, 
unsure, embarassed, shy, fearful, 
bashful, retiring. 


9 

DEPENDABILITY 

Trustworthiness, reiiability, 
responsibility, predictability, in 
responsibilities, promises and 
obligations; punctuality in 
assumed or assigned tasks. 

Not Observed X 


— Low— 

1 2 3 

Relatively undependable, un- 
trustworthy, unreliable, ir- 
responsible in executing respon- 
sibilities, promises, obligations. 
Late with assignments; pro- 
crastinates. Needs continual 
prodding, supervision. 


—Average— 

4 5 6 

Usually trustworthy, responsible, 
reliable, punctual in executing 
most responsibilities, promises, 
obligations. Occasionally needs 
prodding. 


-High- 

7 8 9 

Habitually conscientious, trust- 
worthy, reliable, responsible, 
punctual in executing obligations, 
responsibilities, promises and 
assignments. Has high sense of 
duty; rarely needs prodding. 


10 

LEADERSHIP 

Ability and initiative to direct, 
organize, stimulate, influence 
group action. 

Not Observed X 


-High- 

9 8 7 

An effective leader; forceful, 
capable, inspires confidence. 
Influences, stimulates, organizes 
manages others efforts toward 
group goals. 


—Average— 

6 5 4 

Moderately effective leader in 
organizing, managing, influenc- 
ing others in ordinary situations. 
Usually cannot handle difficult 
groups and situations. 


—Low— 

3 2 1 

A follower, submissive, lacks 
initiative of leader. Unable 
effectively to organize, direct 
others toward group goals. 


Copyright 1963 by August Dvorak, Carroll Mjelde, Alden Nickelson, University of Washington, Seattle, Washington. 
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Great Falls Public Schools 
Great Falls, Montana 



SELF-ESTEEM INVENTORY 

Please mark each statement in the following way: 

If the statement describes how you usually feel, put a check (/) in the column, 
''Like Me/' 

If the statement does not describe how you usually feel, put a check (/) in the 
column, "Unlike Me." 

There are no right or wrong answers. 

Like Me Unlike Me 

spend a lot of time daydreaming. 

I'm pretty sure of myself. 

often wish I were someone else. 

I'm easy to like. 

5. My parents and I have a lot of fun together. 

never worry about anything. 

find it very hard to talk in front of the class. 

wish I were younger. 

9. There are lots of things about myself I'd change 

if I could. 

can make up my mind without too much 
trouble. 



'm a lot of fun to be with. 

get upset easily at home. 

always do the right thing. 

'm proud of my school work. 

Someone always has to tell me what to do. 

t takes me a long time to get used to anything 
new. 

'm often sorry for the things I do. 
'm popular with kids my own age. 
My parents usually consider my feelings, 
'm never unhappy, 
'm doing the best work that I can. 
give in very easily, 
can usually take care of myself, 
'm pretty happy. 

would rather play with children younger than 
me. 



From: Great Falls Public Schools, Great Fails, Montana. 
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26. My parents expect too much of me. 

27. I like everyone I know. 

28. I like to be called on in class. 

29. I understand myself. 

30. It's pretty tough to be me. 

31. Things are all mixed up in my life. 

32. Kids usually follow my ideas. 

33. No one pays much attention to me at home. 

34. I never get scolded. 

35. I'm not doing as well in school as I'd like to. 

36. I can make up my mind and stick to it. 

37. I really don't like being a boy-girl. 

38. I have a low opinion of myself. 

39. I don't like to be with other people. 

40. There are many times when I'd like to leave 
home. 

41. I'm never shy. 

42. I often feel upset in school. 

43. I often feel ashamed of myself. 

44. i'm not as nice looking as most people. 

45. If I have something to say, I usually say it. 

46. Kids pick on me very often. 

47. My parents understand me. 

48. I always tell the truth. 

49. My teacher makes me feel I'm not good enough. 

50. I don't care what happens to me. 

51. I'm a failure. 

52. I get upset easily when I'm scolded. 

53. Most people are better liked than I am. 

54. . I usually feel as if my parents are pushing me. 

55. I always know what to say to people. 

56. I often get discouraged in school. 

57. Things usually don't bother me. 

58. I can't be depended on. 
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CORONADO UNIFIED SCHOOL DISTRICT 
STATE PROGRAM FOR MENTALLY GIFTED MINORS 

PARENT EVALUATION OF PUPIL (Elenrwntary) 



Pupil's Name Parent's Name Date 

Grade 

Please rate your child according to the following scale below, checking the letter 
a, b, c, d, e on the line following each itenn. 

(a) Always (c) About half the time (e) Never 

(b) Usually (d) Seldonn 







a 


b 


c 


d 


e 


1. 


Solves problems independently 












2. 


Demonstrates knowledge in science, social 
studies and literature 












3. 


Is interested in school 












4. 


Uses books and research materials to get 












5. 


Studies independently and plans study time 












6. 


Gets along with others 












7. 


Demonstrates critical thinking ability . . . 












8. 


Is motivated to learn 












9. 














10. 














11. 


Thinks creatively 












12. 


Understands self 












13. 















A similar evaluation will be made at the end of the year^ 



From: Coronado Unified School District, Coronado, California. 
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CORONADO UNIFIED SCHOOL DISTRICT 
STATE PROGRAM FOR MENTALLY GIFTED MINORS 



STUDENT EVALUATION SCALE (Elementary) 



Name Teacher & Room No. Date 

Grade 

Please rate yourself according to the following scale below, checking the letter 
a, b, c, d, e on the line following each item. 

(a) Always (c) About half the time (e) Never 

(b) Usually (d) Seldom 







a 


b 


c 


d 


e 


1. 














2. 


1 know about science, social studies, literature 












3. 














4. 


1 know how to use books and research 












5. 


1 am able to study well alone and budget my 












6. 


1 get along well with other students .... 












7. 


1 am able to consider more than one solution 












8. 


Learning is important to me 












9. 














10. 


1 am able to accept responsibility .... 












11. 














12. 


1 think that 1 can estimate my own strengths 












13. 















A similar evaluation will be made at the end of the year. 



From: Coronado Unified School District, Coronado, California. 
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CORONADO UNIFIED SCHOOL DISTRICT 
MGM EVALUATION FORM FOR USE IN GRADES 7-8 FOR 1974 

TEACHER'S FORM 

Student's Name 

Based on this student's involvement in the Gifted Program and your awareness 
of it, please circle the number which represents your opinion in each of the 
following questions: 

(1 ) to a great extent (2) somewhat (3) very little (4) not at al 
(5) does not apply in my association with this student 



1 . This student was exposed to people in 
selected areas of the business, professional 
and cultural world through one or more field 
trips and/or in bringing in outside speakers. 

Comments: 



2. This student was involved in a variety of 
experiences (projects, reports, group work) 
which seemed to increase his intellectual 
curiosity and aesthetic awareness. 

Comments: 



3. This student is developing basic technical 
skills required for: 

(a) gaining information 

(b) doing independent research 

(c) using the inquiry approach to learning 

(d) preparing special reports 

(e) improving own speech pattern and 
communication skills 

Comments: 



4. This student engaged in activities directed 
toward developing self direction, responsi- 
bility, and leadership skills (small group work 
in classes, projects, student activities etc.) 

Comments: 



1 



2 3 

2 3 

2 3 

2 3 



4 5 

4 5 

4 5 

4 5 



From: Coronado Unified School District, Coronado, California. 
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5. Due to grouping with peers er-i^i ability, 
or projects in my class that are r's idle 
enough to fit the ability and creativity of the 
individual, this student was provided with a 
psychologically safe environnnent where he 
could develop to his fullest potential. 

Comnnents: 
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APPENDIX C 

Sample Instruments for Evaluating 
Inservice Training 



Title 

1 . Post Institute Reaction Sheet 



2. Session Evaluation 



3. Project ASP I R ES - Session 
Evaluation 

4. Pre-Program Reaction Form 



Source Page 

National Summer Leadership 167 
Training Institute of the 
Education of the Gifted and 
Talented 

Movement E Juctition for 1 73 

Handicapped C^^wdren, 
Windsor, Connecticut 

Project ASPIRES, 175 
Hartford, Connecticut 

Project ASPIRES, 177 
Hartford, Connecticut 
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POST-INSTITUTE REACTION SHEET 



Code ID 

INSTRUCTIONS: 

In the following set of questions you are being asked ten aspects of the N/S-LTI-G/T. 
Two rating scales are provided: a quality scale and an importance scale. On the quality scale 
please check the box that appropriately reflects your feelings about the quality of each aspect 
of the LTI. On the importance scale please estimate the relative importance of each aspect 
in contributing to LTI success. For example, if you were rating the quality of the bed springs 
in your room, you might feel they were very good, but of minimum importance to the success 
of the LTI, therefore: 

Quality Importance 

Bed Springs -2 -1 0 -H •t-2 1 2 3 4 5 

Very Poor Barely Good Very Minimum Moderate Maximum 
Poor Acceptable Good Importance Importance 

1.1 Indicate your general feelings about the following aspects of the Summer LTI: 

Quality Importance 

-2 -1 0 +1 +2 1 2 3 4 5 



(a) Quality of instructors □□□□□ □□□□□ 

(b) Quality of room □□□ □□ □□□□□ 
environment ^ 

(c) Effectiveness of □ □□□□ □□□□□ 
location in Squaw Valley 



(d) Quality of your □□□□□ □□□□□ 

accomodations 



(e) Effectiveness of Institute □□□□□ □□□□□ 
management 



(f ) Use of gifted youth at □ □□□□ □ □□□□ 
the institute * 



(g) Length of Summer □□aon □□□□□ 

Institute 



(h) Effectiveness of □□□□□ □□□□□ 

published materials 



(i) 



Appropriateness of 
overall program 



□ □□□□ □□□□□ 
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(j) Extra curricular activities OnnnEII nnn ZD O 
1 .2 What other factors of the LTI were important to you? (List in order of importance) 



2. Do you feel that your insight into the needs and aims of the gifted and talented has been 
improved? 

□ Yes (go to ''Al □ No (go to "5 '7 
A. How much has your insight been improved? 



B. Please comment on what could have been done to improve your insight? 



3. Do you feel that the Summer LTI has helped you gain insight into the politics of 
decision making in education, as it relates to the gifted and talented? 

□ Yes (go to "Al □ No (go to ''B") 

A. How much has your insight improved? (Check one) 

□ □ □ • □ □ 

1 2 3 4 5 



Minimally. Lesr than As expected. More Far more^than 
t3are)v expected very useful than expected, very 
useful but useful expected, useful 

very useful 

B. Please comment on what could have been done to improve your insight. 



Please check one answer concerning the participation of high school students in the LTI. 

□ I feel that the high school students' input contributed significantly to the 
overall effectiveness of the LTI. 

□ I feel that the high school students were a nice touch, but they did not 
contribute much to the input to the LTI. 
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□ 



I feel that the use of high school students was a mistake for the following 
reasons: 



ERIC 



5. What unexpected, but useful information or ability did you obtain from your LTI 
experience? 



6. Did you find the ideas expressed at the LTI to be consistent with your own values? 

□ Always □ Sometimes □ Rarely □ Never 

7. Are you comfortable with your state position paper? 

□ Yes □ No 

8. Did you feel that the state plan you evolved is a workable one? 

□ Yes □ No 

9. Do you feel that this LTI has helped to motivate you towards developing opportunities 
for the gifted? 

□ Yes □ No, I was already motivated □ No, I've lost interest 
10. How good was the organization and education of the Summer LTI? (check one) 

□ Done with great style and precision 

□ Creative in concept but poorly organized 

□ Well organized, but dull 

□ Poorly organized and dull 
A waste of my good time 

□ Other (please specify) 



1 1 . How useful do you anticipate the materials presented at the institute will help you 
set up or improve programs for gifted and talented in your state? 

□ □ m □ ^ ' □ 

-2 -1 0 +1+2 



Very Poor Barely Good Very 
Poor Acceptable Good 



12. In what ways, if any, did the LTI experience change your philosophical position concerning 
the gifted? 



13. How effective was the LTI in terms of your original expectations? 

n n n n □ 

-2 -1 0 +1 +2 



Very Poor Barely Good Very «|gg 
Poor Acceptable Good >' 
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The following list of expectations was generated by you and other persons attending the 
institute through the Pre-lnstitute Reaction Sheet Please indicate your satisfaction (right 
hand column) for each of the items below. 



Expectations relating to state -2 
plan: 

1. Increased insight needed □ 
to develop a comprehen- 
sive state plan. 

2. A specific definitiori for □ 
'^gifted and talented. " 

3. Increased knowledge of □ 
identification procedures. 

4. Increased knowledge of □ 
program planning alter- 
natives (i.e., techniques 

and approaches etc.) 

5. Insight into how to □ 
improve legislation for 

the gifted. 

6. Ideas for teacher training. □ 

7. Kno wiedge of how to □ 
conduct a state assessment 
of present conditions in 
education for G/T youth. 

8. Good criteria for selecting □ 
teachers for the gifted. 

Expectations relating to resources. 

1. Increased knowledge of □ 
what is happening in 
education of the gifted and 
and talented. 

2. Interaction with leaders in □ 
research and programming 
for the gifted. 

3. To develop a closer work- □ 
ing relationship with other 
states in my region. 

4. Increased awareness of □ 
information sources on 

the gifted. 

5. Exchange of ideas with □ 
other states 

Expectation related to follow up: 

1. Insight into the problem □ 
for improved public 
relations. 



Quality 

-1 0 +1 +2 

□ □ □ □ 

□ □ □ □ 

□ □ □ □ 

□ □ n n 

□ □ □ n 

□ □ □ □ 



Importance 

1 2 3 4 5 

□ □ □ □ □ 

□ n n n n 

□ □ □ □ □ 
n n □ □ □ 

n n n n n* 

□ □ □ □ □ 

□ □ a □ □ 

□ □ □ □ □ 

□ □ □ □ □ 

□ □ □ □ □ 

□ □ □ □ □ 

□ □ □ □ □ 

□ □ □ □ □ 

□ □ □ *n n 



□ □ □ □ 

□ □ □ □ 

□ □ □ □ 

□ □ □ □ 
n n □ □ 

□ □ □ 

□ □ □ □ 

□ □ □ □ 
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I feel that the use of high school students was a mistake for the following 
reasons: 



5. What unexpected, but useful information or ability did you obtain from your LTI 
experience? 



6. Did you find the ideas expressed at the LTI to be consistent with your own values? 

□ Always □ Sometimes □ Rarely □ Never 

7. Are ypu comfortable with your state position paper? 

izi Yes □ No 

8. Did you feel that the state plan you evolved is a workable one? 

□ Yes □ No 

9. Do you feel that this LTI has helped to motivate you towards developing opportunities 
for the gifted? 

□ Yes □ No, I was already motivated □ No, I've lost interest 
10. How good was the organization and education of the Summer LTI? (check ohW 

□ Done with great style and precision 

□ Creative in concept but poorly organized 

□ Well organized, but dull 

□ Poorly organized and dull 

□ A waste of my good time 

□ Other (please specify) 



1 1 . How useful do you anticipate the materials presented at the institute will help you 
set up or improve programs for gifted and talented in your state? 

□ □ □ □ □ 
-2 -1 0 +1 +2 



Very Poor Barely Good Very 
Poor Acceptable Good 



12. In what ways, if any, did the LTI experience change your philosophical position concerning 
the gifted? 



13. How effective was the LTI in terms of your original expectations? 

n □ □ n n 

'X -1 0 +1 +2 



Very Poor Barely Good Very ^gg 
Poor Acceptable Good 
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The following list of expectations was generated by you and other persons attending the 
institute through the Pre-lnstitute Reaction Sheet Please indicate your satisfaction (right 
hand column) for each of the items below. 

Quality Importance 

Expectations relating to St' *e -2-1 0 +1 +2 1 2 3 4 5 
plan: 

1. Increased insight needed □□□□□ □ □□□□ 
to develop a comprehen- 
sive state plan. 

2. A specific definition for 00000 OOOOO 
"gifted and talented. " 

3. Increased knowledge of OOOOO O OOOO 
iden tifica tion procedures. 

4. Increased knowledge of OOOO.O OOOOO 
program planning alter- 
natives (i.e., techniques 

and approaches e tc.j 

5. Insight into how to OOOOO OOOOO 
improve legislation for 

the gifted. 

6. Ideas for teacher training. OOOOO OOOOO 

7. Knowledge of how to OOOOO OOOOO 
conduct a state assessment 

of present conditions in 
education for G/T youth. 

8. Good criteria for selecting OOOOO OOOOO 
teachers for the gifted. 

Expectations relating to resources: 

1. Increased knowledge of OOOOO OOOOO 
what is happening in 

educa tion of the gifted and 
and talented. 

2. Interaction with leaders /a? OOOOO OOOOO 
research and programming 

for the gifted. 

3. To develop a closer work- OOOOO OOOOO 
ing relationship with other 

states in my region. 

4. Increased awareness of OOOOO OOOOO 
information sources on 

the gifted. , 

5. Exchange of ideas with OOOOO OOOOO 
other states. 

Expectation related to follow up: 

1. Insight into the problem OOOOO OOOOO 
for improved public 
relations. 
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Date 



Instructor 



SESSION EVALUATION 

I. After responding to each item below, please indicate how important you feel this factor was for the success of the 



session by checking the appropriate box to the left of the question. 



Absolutely 
essential 

□ □ 



□ 
□ 



important 

□ □ 



□ 



□ 



□ 
□ 



Not at all 
important 

□ 



□ 
□ 



□ □ □ □ □ 



□ □ □ □ □ 



□ □ □ □ □ 



□ □ □ □ □ 



□ □ □ □ □ 



1 . Were the objectives of this session made clear? 



2. Were the objectives of this session achieved? 

3. Did the concepts presented in this session 
contribute to your understanding of movement? 



4. Could you see the relevance and applicability of 
the movement concepts presented in this session 
to your functioning in the classroom? 



5. Were the concepts presented in such a way that 
they were easily understood and adopted? 



Very clear 

□ □ 



Could not be 
Hazy deternnined 

□ □ □ 



To a large Not at all 

extent Sonnewhat or could not 
be deternnined 

n n □ □ □ 

Greatly Somewhat Not at all 

□ □ □ □ □ 



Very relevant Not at all 

& applicable relevant or 

Somewhat applicable 

□ □ □ □ □ 



ir-'^.ncepts Sen- Concepts 
easily tances not 
understood unclear understood 

□ □ □ □ □ 



6. Were you given sufficient opportunity to 
practice movements in today's session? 



Sufficient Could use Too much 
opportunity more or or too little 
less practice 

□ □ □ □ □ 



7. Was the physical activity required within your 
capability? 



8. Were you embarassed or uncomfortable as a 
result of activities required in today's session? 



Highly Somewhat 
appropriate strenuous 

□ □ □ 



Too strenuous 
or advanced for 
me to participate 

□ □ 



Very A little Very 
embarrased embarassed comfortable 

□ □ □ □ □ 



I. Answer all of the following which are appropriate to today's session. 
1. How will your general behavior change as a result of today's session? 



2. How will your teaching behavior change as a result of today's session? 



3, How might you use concepts which you learned in today's session in your classroom? 



From: Renzulli J. S., & Ford, B. G. Movement Education for Handicapped Children. Storrs: The University of Connecticut, 1974. 
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Date 



Instructor 



Project ASPIRES - Session Evaluation. 

I. Please check the box that you feel is the best expression, of your opinion about each of the following 
questions. 



1. Were the objectives of this session made clear? 
Comments: 



Could not b« 
V9ry cltar Mazy determined 

n □ □ □ □ 



2. Were the objectives of this session achieved? 
Comments: 



To a large Not at all or could 

extent Somewhat not be determined 

□ □ □ □ □ 



Did the concepts presented in this session contribute to 
your understanding of science? 

Comments: 



Greatly Somewhat Not at all 

□ □ □ □ □ 



Coutd you see the relevance and applicability of the 
science concepts presented in this session to your 
functioning in the classroom? 

Comments: 



Very relevant 
and applicable 



Not at all 
Somewhat relevant or 
applicable 



□ □ □ □ □ 



5. Were the concepts presented in such a way that they 
wi?re easily understoou and adopted? 

Comments: 



Concepts easily Concepts not 

understood Unclear understood 

□ □ □ □ □ 



6. Were you given sufficient opportunity to ask 
questions in today's session? 



Sufficient Not enough 

opportunity Somewhat opportunity 

□ □ □ □ □ 



Comments: 



M. Please comment on any aspects of today's sessiqn that you feel were especially valuable or that you would 
like to see changed in any way. Use the back of this page if you need more space.v 



From: Renzulli, J. S., & Hoffman, F. B. 4n Evaluation of Project ASPIRES. Hartford, Connecticut: The Hartford Public Schools, 1974. 
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NOTE: 

Same instrument is administered 
as a POST PROGRAM REACTION 
FORM so that changes in intended 
gro\/vth can be compared. 

Please do NOT sign your name. 

Part I. Could you explain the following terms of concepts if you were called upon to do so? 

Can you explain or demonstrate: Not at all I could make I could explain I could explain 

a guess or demonstrate or demonstrate 

fairly well very accurately 

1 . how to determine the 

properties of air 

2. how air "behaves" 

3. how to read an anemometer 

4. how to make a rain gauge 

5. how to decode a weather map 

6. how to predict the location 

and phases of the moon — 

7. the descriptive names and 

features of the sun's surface 

8. the descriptive names and 

features of the moon's surface 

9. how to locate man-made 

satellites in the evening sky 

10. how to gather light from solar 

system objects with a telescope 

1 1 . how to identify the five 

properties of minerals 

12. how to classify rocks into 

their three main groups 

13. how rocks are formed 

14. the process of mountain 

building - 

15. the major agents of change of 

the earth's surface — . 

16. the use of a microscope 

17. how to identify various kinds 

of plants 

18. how to sketch lichens 

19. the interdependence of living 

things — — 

20. how \o sketch the growth of 

seeds 

From: RenzulM, J. S., & Hoffman, F.B. An Evaluation of Project ASPIRES. Hartford, Connecticut: The Hartford 
Public Schools, 1974, 
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Project ASPIRES 

PRE PROGRAM REACTION 
FORM 



Can you explain or demonstrate: Not at all 



21. how electricity can be 
produced by chemical and 
magnetic means 

22. how to identify electrical 
components by their symbols 

23. how to make an electrical 
circuit 

24. how to distinguish between 
good and poor conductors 

25. how radio waves carry 
information over long distances 



i could make I could explain I could explain 
a guess or demonstrate or demonstrate 
fairly well very accurately 
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Part II 

1 . Check the following statements that best describe your interest in the subject of science: 
a. It is my favorite subject 

b. It ranks high among the subjects that I like to teach 

c. I neither like or dislike to teach science ^ "* 

d. It ranks low among the subjects that I like to teach 

e. It is my least favorite subject 

2. Check the following statement that best describes your feeling of competency in the teaching 
of science: 

a. I feel extremely competent in this area 

b. I feel somewhat competent in this area 

c. I feel mildly competent in this area 

d. I do not feel at all competent in this area 



3. How often do you ordinarily teach lessons in science? 
a. Every day 

b. Three or four times per week 

c. Once or twice per week 

d. Not at all Yes No 

4. Do you have a clear understanding of the objectives of Project ASPIRES? 

5. Were you sufficiently informed about the purpose of the program prior to 

attending the first session? 

6. Do you think that your students' performance in reading can be improved 

through the study of science? 

7. Do you think that your students' interest in reading can be improved 

through the study of science? 

3. What are your major expectations from participating in this program? 

Please use the back of this page if you need more space. 
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APPENDIX D 

Sample Instruments for Evaluating Various 
Aspects of Program Operation 
and Follow-up Instruments 



Title 

1. Management Evaluation Form 

2. Evaluation of Identification 
Procedures by Selection 
Committees 

3. Self-Evaluation by Teachers in 
Project 

4. Teacher's Appraisal of Creative 
Problem Solving Lesson 

5. Class Behavior Observation 
Checklist 

6. End-of-Year Evaluation 

7. Project Director Questionnaire 

8. Gifted Program-Field 
Evaluation 

9. Instructor Follcw-up 
Questionnaire 

10. Participant Follow-up 
Questionnaire 

1 1 . Movement Education Activities 
Log 



Source Page 

Project Improve 181 
Derby, Connecticut 

Alpha Project for Able Learners 185 
Seattle, Washington 

California Gifted Program 189 

Chicago Public Schools 191 
Chicago, Illinois 

Chicago Public Schools 193 
Chicago, Illinois 

Mentally Gifted Minor Program 195 
Huntington Beach, California 

Illinois Gifted Program 197 

Illinois Gifted Program 203 

Georgia, Governor's Honors 207 
Program 

Georgia, Governor's Honors 213 
Program 

Movement Education for 219 
Handicapped Children 
Windsor, Connecticut 
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PROJECT IMPROVE 
Management Evaluation Form 

Joseph S. Renzulll 
The University of Connecticut 

As part of the overall evaluation of Project I mprove, the evaluators would 
like to obtain some information regarding the effectiveness of the management 
of the project. The purpose of this questionnaire is to obtain feedback that will 
assist the project director in meeting the perceived needs of various groups of 
persons who are connected with the project. This information is being sought at 
the request of the project director for the sole purpose of helping him to determine 
the ways in which he may better serve the project. 

Please do not sign your name to this questionnaire. No attempt will be made 
to identify persons completing the questionnaires. Please return the questionnaire 
in the attached envelope at your earliest convenience. In order. to insure a 
comprehensive evaluation, it is important that all persons complete and return 
their questionnaires. 

Each of the statements below describes a characteristic of a project director 
or activity that might be involved in the management of a project. Please read 
the statements carefully and place an X in the appropriate space according to 
the following scale of values. 

1 . If you have seldom or never observed this characteristic or activity. 

2. If you have occasionally observed this characteristic or activity. 

3. If you have observed this characteristic or activity to a considerable degree. 

4. If you have observed this characteristic or activity almost all of the time. 

5. If you have not had an opportunity to judge this characteristic or activity, 
or if the activity is not applicable to your role in the project. 



From; Renzulli, J. S., & Callahan, C. M. An Evaluation of Project Improve. Storrs: The University of 
Connecticut, 1972. 
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Seldom 
or Never 



Occasional 



5. 



6. 



I . The project director has 
provided me with the type of 
information that Is necessary 
for me to carry out my 
function in the project. 

2. The information has been 
provided in a form that enables 
me to use it in an efficient way. 

3. The information explains the 
type of problem involved 
relevant to the decisions that 
I must make or the action I 
must take. 

4. The project director has 
provided me with enough 
information related to a given 
problem or activity with which 
I am involved. 

The project director makes 
his attitudes and feelings clear 
to me. 

The project director allows me 
to participate in decision 
making when the decisions will 
affect my role or function in 
the project. 

7. The project director feels that 
it is important for me to under- 
stand the reason why certain 
decisions have been made. 

8. When confronted with a 
difficult problem, the project 
director attempts to reach a 
solution that will be at least 
partially acceptable to all 
concerned. 

9. The project director gives 
reasonable consideration to my 
suggestions. 

10. The project director attempts 
to put suggestions made by me 
into practice. 

II. I feel that the project director 
is making the best possible use 
of my talent or ability to 
contribute to the project. 



To a 
Considerable 
Degree 



Almost No Opportunity 
Always To Judge 

Not Applicable 
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Seldom 
or Never 



Occasional. 



12. The project director is just and 
considerate In assigning duties 

to his subordinates. — 

13. The project director is available 
when I want to talk with him. — 

14. The general nnorale of the 
staff is high. 

15. The project director appears 
to be operating from a well- 
organized plan. 

16. I feel that I am sufficiently 
aware of the short and long- 
range direction in which the 
project is heading. 

17. The project director sets 
realistic deadlines and"attempts to 
to see that the deadlines are met. 

18. The project director sees to it 
that the work of persons who 
play different roles in the 
project is well coordinated. 

19. The project director is knowledge- 
able of pertinent details of his sub- 
ordinate's work. 

20. The essential work of the 
project gets done on time. 

21 . The project director is able to 
identify and isolate problems. 

22. The project director is flexible 
and adaptive to changes in events 
and circumstances. 

23. The project director is open- 
minded and willing to change 
or modify his original plans and 
objectives. 

24. When presented with a given 
problem, the project director 
explores a number of possible 
solutions before deciding which 
course of action he will take. 

25. The project director is open to 
new ideas. 

26. The project director is able to 
develop and elaborate upon 
concepts and ideas. 



To a 
Considerable 
Degree 



Almost 
Always 



No Opportunity 

To Judge 
Not Applicable 
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In the space that follows, please comment on any and all aspects about the effectiveness of the 
management of Project Improve. Your frank and honest comments will be of great value in 
bringing about any needed change in the program. 



Please check the space below which applies to you. 

Project Improve Board of Directors 

Project Improve Staff 

Consultant 

Teacher from cooperating school 

Principal from cooperating school 

Member of Placement and Planning Team 

Other 
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ALPHA PROJECT 
FOR ABLE LEARNERS 



Evaluation of Identification 
Procedures By 
Selection Committees 



Prepared by: 
Dr. Joseph S. Renzulli 
University of Connecticut 



Sponsored by: 
Superintendent of Public Instruction 
State of Washington 



1***********************^ ******* 



DIRECTIONS 

Part A: Ratings as an Individual: 

Listed below are the types of information that were used in your school 
district for selecting students for the Alpha Project. Please rate (circle) each 
item according to the usefulness or value of the information in helping you to 
make your decisions. As you rate the items, think only of how the information 
influenced you as an individual rather than how it influenced the entire selection 
committee. If you did not have access to a particular type of information, please 
circle the Not Applicable (NA) column. After you have completed your rating 
for each item, go back over the list and rank the items according to their order of 
importance in influencing your decisions. Assign the rank of "1" to the most 
useful or influential type of information and continue numbering until you have^ 
ranked all items which you have rated. 

As you complete this questionnaire, it will be helpful if you keep a 
copy of the forms and test before you for easy reference. 

Part B : Rating as A Group 

Now do the same thing under the Section labeled Part B, but this time try 
to estimate those factors that were the most influential on the entire selection 
committee. 

Part C: Open- Ended Questions: 

Please answer all of the questions on the last page. 



Name 



Date 



School District 



Position 



ERIC 



172 



185 



Part A: 
Rating as an Individual 



Most 
Useful 



1. Background Information 
Form (Part I) 

A. Summary of School 
Experience 5 

B. Description of 
Parents 5 

C. Description of Family 
Unit 5 

D. Siblings in the HomeB 



3. 



Health Record (Part II) 

A. Data Relevant to Physical 
Development 5 4 

B. Energy Level Rating 5 4 

C. Current Problems or 
Handicaps 5 4 

D. History of Illnesses 

or Problems 5 4 

E. Health Tests and 
Measurements 5 4 

F. Results of Medical 
Examinations 5 4 

Screening and Nomination Form 
(Part Mi) 

A. Academic Achievement 
Tests 5 



4 
4 



B. Group Ability Tests 5 

C. Individual Intelligence 
Tests 5 



D. Other Tests or 
Examinations 5 4 3 

E. | Rating of Intellectual Functioning 

(Items 1-26) 5 4 3 

F. Interest/Performance/ 
Capability Rating 
(Between Items 26 

and 27) 5 4 3 



Least 
Useful 



Rank 



1 NA 



1 NA 



2 1 NA 
2 1 NA 



1 NA 
1 NA 

1 NA 



2 1 NA 



2 1 NA 



2 1 NA 



2 1 NA 

2 1 NA 

2 1 NA 

2 1 NA 

2 1 NA 



2 1 NA 



Part B: 
Rating as a Group 



Most 
Useful 



Least 
Useful 



5* 4 3 2 

5 4 3 2 

5 4 3 2 

5 4 3 2 



1 NA 

1 NA 

1 NA 

1 NA 

1 NA 

1 NA 

5 4 3 2 1 NA 

5 4 3 2 1 NA 

5 4 3 2 1 NA 

5 4 3 2 1 NA 



5 4 3 2 1 NA 

5 4 3 2 1 NA 

5 4 3 2 1 NA 

5' 4 3 2 1 NA 

5 4 3 2 1 NA 



Rank 



1 NA 
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Part A: 
Rating as an Individual 



Most 
Useful 



G. Physical Development 



10. Other (specify) . 



Least 
Useful 



Rank 





(Items 27-32) 


5 


4 


3 


2 


1 


NA 




H. Social Development 
(Items 33-41) 


5 


4 


3 


2 


1 


NA 




1. Emotional Development 
(Items 42-53) 5 


4 


3 


2 


1 


NA 


4. 


Parent Inventory 
(Part IV) 


5 


4 


3 


2 


1 


NA 


5. 


Pupil Inventory 


5 


4 


3 


2 


1 


NA 


6. 


Pupil Rating 


5 


4 


3 


2 


1 


NA 


7. 


Preference for Working 
Conditions 


5 


4 


3 


2 


1 


NA 


8. 


Value Ranking 


5 


4 


3 


2 


1 


NA 


9. 


Other (specify) 
















5 


4 


3 


2 


, 1 


NA 



1 NA 



Part B: 
Rating as a Group 



Most 
Useful 



Least 
Useful 



4 
4 
4 



3 
3 
3 



1 NA 
1 NA 
1 NA 



5 


4 


3 


2 


1 


NA 


5 


4 


3 


2 


1 


NA 


5 


4 


3 


2 


1 


NA 


5 


4 


3 


2 


1 


NA 


5 


4 


3 


2 


1 


NA 


5 


4 


3 


2 


1 


NA 


5 


4 


3 


2 


1 


NA 



Rank 
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Open-Ended Questions 

In the space below list or describe any information that was not available 
but that you would hgve liked to have had in making your decisions about 
student selection for the Alpha Project. 



What changes would you like to make in the screening and identification 
process? 



In the space below please comment about your overall satisfaction and/or 
dissatisfaction with the total screening and selection process. We are 
especially interested in both positive and negative reactions. Use the back 
of this page if you need more space. 



SELF-EVALUATION BY TEACHERS IN PROJECT 



Teacher Date 

One objective of the State Study is to increase teacher confidence and 
competence in teaching gifted children. We would, therefore, like your subjective 
evaluation on your current feelings in this area. 

Please evaluate yourself on the following items, using a, b, c, d according to the 
sacle below. 

(a) Diminishing (c) Increasing 

(b) About the same (d) Marked increase 

1 . Strength and skill as a teacher . ' ^ 

2. Knowledge of subject areas 

3. Appreciation of "gifted" pupils - 

4. Enthusiasm for your particular classroom plan in the State Study . 

If no reaction, leave blank. 

1. What changes would you suggest in your particular educational plan? Why? 



2. In your opinion, what are the values of your particular plan in the State 
program? 



3. How has participation in the State Study helped you? 



4. List any problems created by your participation in the State Study. 



From: Simpson, R. E., & Martinson, R: A. Educational Program for Gifted Pupils. Sacramento, 
California: California State Department of Education, 1961. 



TEACHER'S APPRASIAL OF 
CREATIVE PROBLEM SOLVING LESSON 



DemonstratoV School Observer 

Date Room Number Grade Level 



PLEASE INDICATE WITH AN X THE EXTENT TO WHICH YOU AGREE WITH EACH OF 
THE FOLLOWING STATEMENTS. 



Unable Little Somewhat Much A great 

to deal 
Observe 



1. Do you think a problem was structured so 
as to lead to the discovery of a new concept 
or understanding? 

2. Were you able to observe a diversity of 
pupil responses in the initial stages of the 
problem? , 

3. To what extent do you feel that the pupils 
drew upon personal past experiences to 
solve the problem? 

4. To what extent do you think the teacher 
"structured" the discussion? 

5. Do you feel the pupils understood that there 
is always more than one answer or way to 
arriving at the solution to a problem? 

6. Was a classroom climate established whereby 
each pupil would feel free to contribute to 
the class discussion? 

7. Did pupils contribute most of the information 
and ideas that were necessary to arrive at a 
solution to the problem? — 

8. Did pupils discover meaningful new relation- 
ships between the information and ideas they 
contributed to the problem? — 

9. Were the pupils allowed and encouraged 
to react to other pupil's responses? — 

10. Did pupils vie with each other to answer 
questions? 

11. Do you feel the majority of the pupils 
"learned" the concept? . 

12. Write any comments you may have concerning the demonstration lesson. 
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From: Chicago Public Schools. In-Service Training Program for the Promotion of Creative Problem- 
solving (Second Revised Edition). 
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CLASS BEHAVIOR OBSERVATION CHECKLIST 
Teacher's iMame Room No.. Grade 

School Date 

THE FOLLOWING QUESTIONS HAVE ONE OR MORE ANSWERS. PLEASE CHECK 
THE ANSWER(S) WHICH BEST DESCRIBE CLASS' BEHAVIOR. 

1. In leading your class through the beginning stages of a creative problenn-solving 
activity, were you able to: 

A. present the problem to the class and note their reactions. 

B. demonstrate the value of divergent views of the same problem by deliberately 

provoking disagreements. 

C. accept novel and unconventional solutions in an attempt to get as many 

solutions as possible. 

D. encourage group agreement, but still permit unusual responses if they were 

pertinent to the problem. 

E. encourage group agreement, on the description or perception of the problem. 

F. Other. Explain 



2. If a pupil response was unclear or definitely unrelated did you: 

A. clarify the pupil's response. 

B. ask the pupil to clarify it. 

C. ask another pupil to clarify it, 

D. simply acknowledge the response and go on. 

E. ignore the response and go on. 

F. Other. Explain. 



3. Did you direct the class toward unity in defining and stating the problem by: 

A. utilizing praise to obtain the desired response. 

B. insisting that the class pay attention to all the evidence. 

C. pointing out overlooked evidence until one or more pupils offered a logical, 

consistent explanation of the problem. 

D. Other. Explain. 



4. Were you able to prevent all or part of the class from forming a premature opinion 
or from adopting one child's opinion by: 

A. not permitting any one child or yourself to dominate the discussion. 

B. encouraging group uncertainty and group interactions. 

C. encouraging as many responses as possible from as many pupils as possible. 

D. Other. Explain. . 



From: Chicago Public Schools. In-Service Training Program for the Promotion of Creative Problem-Solving 
(second Revised Edition.) 
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Most groups have one or two children who do most of the answering. Were you 
able to get different pupils to respond to the problem by: 

A. encouraging a child even if he was on the wrong track? 

B. relating the problem to the children's own background. 

C. deliberately calling on the^npntcontributing members, of the class. 

D. giving up, realizing that all pupils cannot contribute at all times, or that each 
group has its leader. ^ 

E. Other. Explain. 



Were you able to get the children to see how the new concept related to their 
personal lives by: 

A. pointing out the concept's relationship to their daily lives. 

B. asking pupils to find the concept's relationship to their daily lives. 

C. giving up, realizing that some knowledge is "booklearning," unrelated to daily 
lives. 

D. Other. Explain. 



Were you able to get the children to examine each other's ideas and yours by: 

A. pointing out to them that an idea which has been held true for a long time is 
sometimes erroneous in light of new information. 

B. pointing out that no one person can know everything. 

C. Other. Explain. 
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OCEAN VIEW SCHOOL DISTRICT 
MENTALLY GIFTED MINOR PROGRAM 
END OF YEAR EVALUATION 

(To be filled out by May 31, 1974. Copy to be sent to the District MGM Coordinator, 
Area Administrator's Office.) 



School Date Principal or MGM Coordinator 

This evaluation instrument is designed to be an action-planning instrument for program 
improvement. If your ?nswer to a question is "yes", plan in your school's next year's 
educational plan how to do an even better job. If your answer is "no", and your school 
believes the idea to be a good one, put the idea in your next year's educational plan for 
implementation. 

1. Does your school have an MGM Coordinator(s) appointed by the principal? 



Yes No 
2. Do you include'the MGM Program in your school's educational plan? 



Yes No 
3. Is there a written description of your school's MGM Program? 



Yes No 

Does your school have an MGM program evaluation and program improvement 
committee made up of staff and/or parents that meets at least periodically? 



Yes No 

Do you send out a notice at the beginning of each semester asking for teacher 
referrals 6n potentially gifted students so that they can be screened for the 
MGM Program? 



Yes No 

Do you include with this notice the district's guide sheet to assist the teachers 
in identification of students to be referred for screening? 



Yes No 

Is each student admitted to the program approved by the Admissions and 
Dismissal Committees? t 



Yes , No 

8. Does your Admissions and Discharge Committee evaluate and recommend 
placement for each MGM Student at the end of each year? 



Yes No 



From: Ocean View School District, Huntington Beach, California. 
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9. Does your Admissions and Discharge Committee l<now that once an MGM youngster 
is identified a program must be created to meet his/her needs? (An MGM youngster 
cannot be removed from the program without a signed parent approval letter to 
accompany his discharge.) 



Yes No 



196 
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School E 
Form Five 

PROJECT DIRECTOR QUESTIONNAIRE 

The following questions may be answered jointly by the project director and 
other administrators if desired. 

I. On a separate page, briefly describe the history of the Gifted Project in School 
E. Use any sources you believe to be pertinent, e.g., school board minutes, 
newspaper articles, financial records, final reports. 

II. Write a statement of the goals and rationale for the Gifted Project. 



From: Sjogren, 0., Hopkins, T., & Cooler, D. Evaluation Plans and Instruments: Illustrative 
cases of Gifted Program Evaluation Techniques, Center for Instructional Research and 
Curriculum Evaluation. Urbana-Champaign: University of Illinois. 
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III. 



What office space, storage, and classroom space were provided for the project? 



IV. What administrative activities preceded the implementation of the program 
after the funds were available? e.g., recruitment of teachers, scheduling, 
presentation to staff. (This question might be answered by an outline of 
steps taken.) 



198 
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V. 



a) Briefly deKribe the rationale for selecting teachers for the program. 



ERIC 



b) What types of problems were encountered in the selection process? 



VK Describe any adjustments made in teaching and administrative loads for the 
Gifted Project staff. 
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VII. Describe the provisions for the project staff to work together. 



VIII. List the ways in which the concepts of the Gifted Program have been p»'esented 
to others (staff, community/school board, students, etc.) 



IX. List any situations which you have observed that would reflect either opposition 
to, or support for the Gifted Program. 
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X. Training, for the teaching of the gifted, which has been available to teachers. 



Title of course 
or workshop 


Sponsored by 


Staff who 
attended 


Dates 


Length of 
training 













XI. List other teachers who have expressed an interest in this project. In a 
few words describe their interest, e.g., want to teach, want to attend a 
workshop, have assisted the staff. 
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XII. Does the Gifted Program require any special scheduling procedures? 

Yes No 

If "Yes," briefly describe. 



XIII. Provide a copy of the school schedule. 
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Evaluation Team 



Staff Form 
Rev. 7/73 



(Team Reporter) 

Date of Visitation. 



GIFTED PROGRAM - FIELD EVALUATION 

Program Title Program Number. 

Program Site Dist Area 

Contact= Phone 

Teacher 

Starting date of Prngram Number of Students 

by grade 



DID NOT 

PROGRAM DESCRIPTION OK PROBLEM OBSERVE 



1 . Are the depth and focus of activities in the 
program generally such that they meet the 
special needs of gifted children? 








2. Are children placed out of certain aspects of 
the regular program as a consequence of gifted 
program participation? 








3. Are the program's activities compatible with 
the program's objectives? 








PROGRAM IDENTIFICATION 

4, Do the identification procedures correspond 
to those stated in the current proposal? 








5. Is student identification based primarily on 
objective data? 








PROGRAM MECHANICS 

6. Does the number of students in the program 
approximate the number of students stated 
in the current proposal? 








7. Are the students involved in the program at 
least 150 minutes per week? 








8, Does the program a-llow for an amount of 
student participation in setting objectives, 
planning activities, and evaluation progress? 








9. Is the interaction between teacher and 
students generally appropriate to the 
program objectives? 









From: State of Illinois Gifted Program, Office of the Superintendent of Public Instruction, Springfield, 
Illinois. 
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DID NOT 





OK 


PROBLEM 


OBSERVE 


lu. uu6b sTuaeni invoiverneni in ine program 
activities seem generally to be high? 








1 1 . Are special counseling and/or guidance 

pruvioiunb maae avaiiaDie lo an STUuenis in 
the gifted program? 








PROGRAM EVALUATION 

1 ^. lo a oyoLclIlaLiU pidll Ul evaiUdiiun Uciny Uoeu 

to assess the program? 








13. Does the evaluation include measuring 
growth? 








14. Does the evaluation provide necessary 

1 1 1 1 Ul 1 1 Id LIUl 1 LU doocoo LI Ic c 1 1 LI Vcf icbo U I ine 

program? 








PRnnRAM AnMIMIQTR ATinM 
• Hv^ljnMIVI ML^IVIilMio 1 nM 1 i^iM 

15. Were the teachers in the program involved in 
developing the current proposal? 








1 o. uu ine leacners in ine program nave copies 
OT the current proposal? 








17. Are materials and services being purchased 

MCUcoodi y LU Lllc dULlViLico Wniun LUilipr loc 

the program? 








18. Do the teachers in the program have access to 

^ll matpria!<5 niirphaQpH cnopif ipalK/ fnr tho 

all lllaLdluiO ^lUIVrfl luDCU OLlCUIIIUdiiy lUl LlIC 

program? 








19. Are program funds being utilized mainly for 
program development rather than program 
maintenance? 








20. Is budget record kept of current balance? 








21. Are teachers aware of balance of funds 
dHuudLcu anu ucauinicb Tor suomission OT 
requisitions? 








22. Do school clerks have special ordering 
instructions for materials and services for 

yiiLcu piugiaiiidf 








PROGRAM SERVICES 

23. Are materials and services ordered for the 
program received within a reasonable length 
of time? 








24, Are you receiving the services and consultive 
help necessary to assist you in implementing . 
your program? 
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25. Indicate in which of the following areas you would like to receive additional 
assistance from the Gifted Program staff, such as: 

Identification of gifted children 

Developing program objectives in behavioral terms 

Program evaluation 

Teaching strategies 

Other: 



EVALUATOR'S AUDIT SUMMARY 
(Include explanation for any item checked "problem") 
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EVALUATOR'S RECOMMENDATIONS (Please number) 



FOLLOW-UP ACTION 



Copies to: 



(Team Reporter - Signature) 
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GOVERNOR'S HONORS PROGRAM INSTRUCTOR FOLLOW-UP 

QUESTIONNAIRE 

The Governor's Honors Program is a relatively high-cost project serving only 
a limited proportion of the high school students in the state of Georgia. For 
this reason the Governor and members of the State Department of Education 
are concerned that the program be maximally effective. Since you have partic- 
ipated in the program as a staff member, your opinion about the program 
would be mosf'valuable in our current evaluation of the program and, in turn, 
to state officials in making decisions concerning the program. Your cooperation 
in filling out the questionnaire, making helpful suggestions, and returning it in 
the enclosed envelope will be greatly appreciated. 

Sex: M F Years of Experience 

1. What year(s) were you a staff member of the Governor's Honors Program? 
19 

2. Check your teaching or specialization area. 

1 . Art 7. Science 

2. Drama 8. Social Science 

3. English 9. Physical Education 

4. Foreign Language 10. Counseling 

5. Mathematics 11. Other Area 

6. Music 

Following is a list of factors which are important in effective operation of the 
Governor's Honors Program. You are asked to rate the program on each of the 
factors by checking one of the spaces at the right of each statement. Use what 
you would consider as the ideal program as a standard of excellence in making 
your ratings. 

If the program was EXTREMELY POOR with respect to the factor, check 
space 1 . 

If the program was BELOW AVERAGE, with respect to the factor, check 
space 2. 

If the program was ACCEPTABLE, with respect to the factor, check space 

3. 

If the program was ABOVE AVERAGE, with respect to the factor, check 
space 4. 

If the program was EXCELLENT, with respect to the factor, check space 5. 

Following each rating is a space you might use to suggest changes related to 
the factor which you think would improve the program, or to describe various 
influences. 

3. Value of overall objective of the program 1 — 2 — 3 4 — 5 — 

Suggested Changes: 



From: Payne, D. A. Evaluation of the State of Georgia's Governor's Honors Program. 
Athens: University of Georgia, 1972. 



192 



4. Accomplishment of your own instructional 
objectives as a staff member 

Suggested Changes: 



5. Suitability of the methods by which students 
are selected to participate in the program 

Suggested Changes: 



6. Suitability of your instructional methods for 
G HP students 

Suggested Changes: 



7. Degree to which the program enabled you to 
make contributions to or initiate changes in your 
local school program 

Suggested Changes: 



8. Contributions of program toward making a 
positive change in student attitude toward 
learning. 

Suggested Changes: 



9. Suitability of facilities and equipment available 
for teaching GHP students 

Suggested Changes: 
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10. Influence of facilities andiequipment availability 1 2 3 4 5- 

on effective teaching of GHP students 

Ratio of Influence: 



1 1 , Influence of progrann on changes in your 1 2 3 4 5 

instructional nnethods during the program 

Ratio of Influence: 



12, Your ability to nriaintain an ideal classroom 1 2_ 3 4 5 

atmosphere for GHP students 

Suggested Changes: 



13. Influence of GHP on your local selection of 1 2 3 4 5. 

subject matter content 

Suggested Changes: 



14. Influence of program in making significant 1 2 3 4 5 

change in your instructional method upon 
returning to your local situation 

Ratio of GHP Influence: 



15. The effectiveness of the administration of the 1.>._ 2 3 4 5. 

program 

Suggested Changes: 




16. The effectiveness of the organization of the 1- 2 3 4 5 

program 

Suggested Changes: 
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17. The degree to which your experience as a G HP 1 — 2 3 4 5. 

staff member influenced you to have a more 
positive attitude toward gifted students 

Suggested Changes: 



18. Usefulness of special events (speakers, concerts , 1 2 3 4 5. 

etc.) 

Suggested Changes: 



19. Usefulness of the seminars 1 2 3 4 5 

Suggested Changes: 



20. Opportunity for students to interact with each 1 2 3 4 5. 

other 

Suggested Changes: 



21. Opportunity for students to interact with 1 2 3 4 5 

teachers 

Suggested Changes: 



22. Student perception of overall value of GHP 1 2 3 4 5 

Reasons: 



23. What two things do you think are most beneficial about the program? 
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24, What two things were least beneficial or in the greatest need of change with 
regard to the program? 



Additional Comments: 
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GOVERNOR'S HONORS PROGRAM PARTICIPANT FOLLOW-UP 

QUESTIONNAIRE 

The Governor's Honors Prog.'am is a relatively high-cost project serving a 
limited proportioi: of the high school students in the state of Georgia. For 
this reason the Governor, State Superintendent of Schools, G HP personnel, 
and members of the State Department of Education are concerned that the 
program be maximally effective. Since you are a former participant in the 
program, your opinion about the program would be most valuable in our 
current evaluation of the program and, in turn, to state officials in making 
decisions concerning the program. Your cooperation in filling out the question- 
naire, making helpful suggestions, and returning it in the enclosed envelope 
will be greatly appreciated. 

Sex: M F Age: 

1 . What year did you attend the Governor's Honors Program? 1 9 

2. Check your area of nomination 

1 . Art 5. Mathematics 

2. Drama 6. Music 

3. English 7. Science 

4. Foreign Language 8. Social Science 

3. Check the statement(s) which is (are) applicable to you 
01 . I am still attending high school. 

02. I have been graduated from high school, but have not attended 

and do not plan to attend college. 

03. I attended college but did not obtain a bachelor's degree. 

. 04. I am currently attending college working toward a professional 

(bachelor's) degree. 

05, I am currently attending a non academic school or college 

working toward a technical profession. 

06, I currently hold a bachelor's degree. 

07, I currently hold a degree from a technical institution. Type 

of degree 

08. I am currently pursuing a graduate degree. Type of degree 

09. I currently hold a graduate degree. Type of degree 

10. I am currently employed (full time). Type of work 

Following is a list of factors which are important in effective operation of 
the Governor's Program. You are asked to rate the program on each of the 
factors by checking one of the spaces at the right of each statement. Use what 
you would consider as the ideal program as a standard of excellence in making 
your ratings. 



From: Payne, D. A. Evaluation of the State of Georgia's Governor's Honors Program. 
Athens: Univb.sity of Georgia, 1972. 



If the program was EXTREMELY POOR with respect to the factor, check 
space 1 . 

If the program was BELOW AVERAGE with respect to the factor, check 
space 2. 

If the program was ACCEPTABLE with respect to the factor, check 
space 3. 

If the program was ABOVE AVERAGE with respect to the factor, check 
space 4. 

If the program was EXCELLENT with respect to the factor, check space 5. 

Following each rating is space you might use to suggest changes related to 
the factor which you think would improve the program, or allow you to describe 
influences. 
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4. The degree to which the program was beneficial 1 2 3 4 5. 

in your subsequent academic course selection 

Describe Nature of GHP Influence: 



5. The degree to which the program influenced your 1 2 3 4 5 

decision to attend college 

Describe Nature of GHP Influence: 



6, The degree to which the program was beneficial 1 2 3 4 5 

in helping you choose a college major (Omit, if 
you did not attend college) 

Describe Nature of GHP Influence: 



7. The degree to which the program was beneficial 1 2 3 4 5 

in helping you choose a vocation 

Describe Nature of GHP Influence: 
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8. Suitability of the method or methods by which 1 2 3 4 — 5^ 

participants were selected 

Suggested Changes (e.g., what criteria should be 
employed.): 



9. Suitability of the instructional methods for GHP 1 2 3 4 5 

students 

Suggested Changes: 



10. Appropriateness of the administration of the 1 2 3 4 5 

program 

Suggested Changes: 



1 1. Influence which the program had on your 1 2 3 4 5 

ability to make contributions to or initiate 
changes in your local school program 

Suggested changes in the program which would have 
enabled you to more effectively initiate changes in 
or make contributions to your local school program: 



12. Contributions the program made toward a positive 1 2_ 3 4_ 5. 

change in your attitude toward learning 

Nature of GHP Influence: 



13. Helpfulness of the counseling program 1 2 3 4^ 

Suggested Changes: 
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14. Effectiveness of the physical education program 1 2 3 4 5- 

in teaching you gannes or other recreational 
activities which you did not have the opportunity 
to learn in your high school 

Suggested Changes: 



15. Usefulness of the seminars " , 1 2 3 4 5- 

Suggested Changes: 



16. Usefulness of special events (speakers, concerts, 1 2 3 4 5-. 

etc.) 

Suggested Changes: 



17. Opportunity for interaction with other students 1 2 3 4 5- 

Suggested Changes: 



18. Opportunity for interaction with teachers 1 2 3 4 5. 

Suggested Changes: 



19. Your overall rating of the program in terms of 1 2 3 4 5. 

fulfilling your immediate needs at the time you 
participated 

Suggested Changes: 



216 




200 



20. Overall rating of the program in terms of 
fulfilling your ultimate goals 

Suggested Changes: 



21. Degree to which the program objectives were in 1 — 2 — 3 — 4 — 5, 
agreement with your personal objectives 

Suggested Changes: 



22. Extent to which you mastered the objectives 1 — 2 — 3 — 4 — 5 

of the program 

Suggested Changes: 



23. Extent to which the program contributed to 1 — 2 — 3 — 4 — 5 

your mastery of the program objectives 

Suggested Changes: 



24. What two things were most beneficial about the program? 



25. What two things were least beneficial or in the greatest need of change with 
regard to the program? 



26. What Honors, Awards, Scholarships, Fellowship Grants, or Special 
Recognitions have you received since you were a GHP participant? 



o 
o 

Q. 

_> 
0) 

E 

X 
LU 



0) 
O) 
CO 
k_ 
0) 

> 

< 

CO 



SS 
a 

o 
o 

< 



0) 

> 

< 

> 

o 



c 
o 

X 



3_ 4_ 5- 



201 



217 



MOVEMENT EDUCATION ACTIVITIES LOG 



Name School 

Type(s) of Students Age Range - 

Number of students in your group 



Directions: The purpose of this LOG is to provide a record of (1 ) the types 
of Movement Education Activities that you are conducting with your class, 
(2) the frequency with which such activities are pursued, and (3) the students' 
general reaction to each activity. We hope that the LOGS will give us some 
direction in providing future workshops that are designed for various types of 
exceptional children, and therefore, it is important for all LOGS to be kept 
consistently and conscientiously. 

Please, keep your LOG up-to-date and try to provide a descriptive name for 
each type of activity in which you engage. If the "General Reaction of Students" 
is mixed, please, indicate this in the "Comments" section. 

" Tf you have a record or rough idea of activities that were pursued prior to 
receiving the LOG, please list the activities according to your best recollection. 

" The LOGS will be collected during the Spring of 1974 and, at that time, we 
will ask you to summarize the general areas of positive and negative reactions. 



From: RenzulIi,J.S.,&Ford,B.G. Mo vemen t Educa Hon for Handicapped Children. 
Storrs: The University of Conneticut, 1974. 



Name 



School 



Date 



General Reaction of Students 
(Check one) 



Date Type of Activity >^ <^ ^ <^ Comments 
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Appendix E 
Sample Evaluation Contract 



CONTRACT FOR PROFESSIONAL SERVICES 

BETWEEN 



AND 



THIS AGREEMENT made by and between the City of 

, hereinafter designated 

as the City and '. , 

hereinafter designated as the Evaluator. 

ARTICLE 1 

Description of Services to Be Performed. The Evaluator shall upon the 
basis of his training and experience enter upon the elementary and/or secondary 

level of the 

Public School System and more particularly at the following schools and/or 
Programs: 



and conduct a survey, analysis, and study of the same in action in order to 
evaluate and assess the efficiency and productivity of same in action, and upon 
the completion of the said survey, study, and analysis, to file with 



a comprehensive and detailed written report thereon in 

( ) copies, setting forth the results, 

conclusions, and findings thereon together with comprehensive recommendations 
in respect thereto. 

ARTICLE 2 

Information to be Furnished by City. The City shall furnish to the Evaluator 
whatever information, data, and statistics are in its possession which may be 
useful to the Evaluator in carrying out his survey under this contract. 

ARTICLE 3 

Time for Commencement and Completion of Services. The Evaluator shall 
commence his services under this agreement forthwith upon the acceptance of 
this contract by the City and shall complete his services and file his final report 
not later than thirty (30) calendar days after the termination date of the contract. 

Progress reports shall be submitted every days and cover 

completely all the services performed since the date of the last progress report. 
Upon receipt of the final report, the Evaluator shall be required to amplify in 
writing any phase of the report which may not, in the opinion of _ 
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be sufficiently comprehensive. The Evaluator and the City shall perform their 
respective activities and responsibilities in accordance with the dates outlined 
in the attached Statement of Responsibilities. 

ARTICLE 4 

Compensation to Be Paid by the City. It is expressly agreed and understood 
by the parties hereto that in no event shall the City pay the Evaluator as full 
compensation for everything furnished or done by or resulting to the Evaluator 
in carrying out this agreement a total sum in excess of $ 

The Evaluator shall also be reimbursed for "travel expenses" if previously 
authorized in writing, up to the maximum amount of $ 

ARTICLES 

Time for Payment to Evaluator. Application for payment shall be made in 
accordance with the attached work phase and Payment Schedule. Each application 
shall show the name of the Evaluator and/or his employees engaged in the study, 
the number of hours (or days) worked, and his (or their) total wages at the 
contract rate{s), such abstract to be sworn to by the Evaluator. 

ARTICLE 6 

" Discontinuance of Service. The contract may be terminated by either party 
at any time by a notice in writing duly mailed or delivered by one party to the 
other. In the event of contract termination, Evaluator will be paid in accordance 
with Article 4 for all of his services duly performed up to the date of discontinuance. 

ARTICLE 7 

Laws and Regulations. This contract is subject to all laws of the State of 

^ , applicable to the administration of public 

schools and likewise to all Rules and Regulations of the School Committee of 

the City of : , : , as amended to 

date. A copy of the aforesaid Rules and Regulations is on file in the office of the 
Secretary of the School Committee. 

ARTICLES 

Final Release. In consideration of the execution of this contract by the 
City, the Evaluator agrees that, simultaneously with the acceptance of what the 
City tenders as the final payment by it under this contract, Evaluator wil! execute 
and deliver to the City an instrument under seal releasing and forever discharging 
the City. of and from any and all claims, demands, and liabilities whatsoever of 
every name and nature, both at law and in equity, arising from, growing out of, 
or in any way connected with this contract, save only such claims, demands, and 
liabilities as are expressly excepted in said instrument. 
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ARTICLES 

Assignment. Neither the School Committee nor the Evaluator shall assign 
or transfer the respective interests in this contract without the prior written 
consent of the other. 



Evaluator 

Title: (If corporation affix seal) 
Approved: 

Business Manager 

Accepted: '. : 19. 

THE SCHOOL COMMITTEE OF THE CITY OF 



Chairman of the School Committee 



STATEMENT OF RESPONSIBILITIES 

The Evaluator and the City will perform their respective activities and 
responsibilities in accordance with the following Statement of Responsibilities: 



1 . Management of the Project 

2. Determine and Write Objectives 

3. Select Evaluative Criteria 

4. Select and Construct Measuring Instruments 

a. Standardized Instruments 

b. Questionnaires and Rating Scales 

c. Teacher-made Instruments 



d. Interview Schedules 

5. Purchase Instruments 

6. Administer Instruments 

7. Conduct Interviews 

8. Score Instruments 

a. Standardized Instruments 

b. Questionnaires and Rating Scales 

c. Teacher-made Instruments 

9. Data Tabulation, Summary and Analysis 

10. Report Results and Conclusions, Make 
Recommendations 

1 1. Integration of Evaluation Report into 
Final Report 



City 

Responsible 
Responsible 
Responsible 

Suggest 
Suggest 
Responsible 

Suggest 

Responsible 

Responsible 

Arrange 

Responsible 

Responsible 

Advise 
Responsible 



Evaluator 

Consult 
Consult 

Responsible 

Responsible 

Propose, 
advise and 
confirm 

Responsible 

Consult 

Monitor and 
Train 

Responsible 

Consult 
Responsible 
Consult 
Responsible 

Responsible 

Consult 
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WORK PHASE AND PAYMENT SCHEDULE 

The Evaluator and the City will perform their respective activities in accordance 
with the following schedule: 

Phase I July 1,1 9 to October 31, 19 

1. The Evaluator collaborates with the City and project staff in determining 
the overall evaluative design and the plan for implementing the design. 

2. The City formulates general, intermediate, and specific behavioral objectives 
which will be submitted to the Evaluator for review. 

3. The Evaluator proposes, advises, and confirms the selection of evaluative 
instruments. 

4. The Evaluator submits to the City a complete evaluation design. 

5. Selected pre-project instruments are administered by the City. 

6. The Evaluator gathers input information for questionnaire and rating scale 
design. 

7. The Evaluator pre-trains interviewers and project staff in gathering evaluative 
data relevant to their respective responsibilities. 

8. Payment due October 31,1 9 $ . 

Phase II November 1, 19 to January 31, 19 

1. As the program evolves, the City cor>tinues to formulate objectives and the 
Evaluator assists teachers in the construction of evaluative instruments. 

2. The Evaluator prepares questionnaires, rating scales and interview schedules; 
gathers and analyzes data; and provides periodic feedback to the City. 

3. The Evaluator tabulates and summarizes pre-project data for statistical 
analysis. 

4. Payment due January 31 , 1 9 $ 

Phase III February 1, 19 to April 30, 19 

1. Steps 1 and 2 in Phase II are repeated. 

2. Evaluator prepares end-of-project (summative), evaluates instruments. 

3. Payment due April 30, 19 $ 

Phase IV May 1 , 1 9 to June 30, 1 9 

1. City administers summative evaluative instruments. 

2. Evaluator tabulates, summarizes, and analyzes evaluative data. 

3. Evaluator collaborates with City on writing the final report. 

4. The Evaluator submits an evaluation report to the City. 

5. Payment due June 30, 19 $_ 



208 



Title VI of the Civil Rights Act of 1964 states: 



"No person in the United States shall, on the ground of race, color, or national 
origin, be excluded from participation in, or denied the benefit of, or be 
subjected to discrimination under any program or activity receiving federal 
financial assistance/' 

Therefore, EPDA programs must be operated in compliance with this law. 
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